2024-09-17T18:07:42.288 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T18:07:42.300 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T18:07:42.385 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909236 branch: wip-bharath10-testing-2024-09-17-1348 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: skanta@redhat.com first_in_suite: false job_id: '7909236' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-bharath10-testing-2024-09-17-1348 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 workunit: branch: wip-bharath10-testing-2024-09-17-1348 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 3421 sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath10-testing-2024-09-17-1348 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBELJBFNzTm+TV3b/m57iBFjHHCJ7KTKz5bnlRZ+/6kDhc5kuTIcJzRjzx29EODKneqTHqbdIZuwVtABVINGMqMQ= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_15:01:33 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T18:07:42.386 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa; will attempt to use it 2024-09-17T18:07:42.386 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks 2024-09-17T18:07:42.386 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T18:07:42.388 INFO:teuthology.task.internal:Checking packages... 2024-09-17T18:07:42.408 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '1889a165b5128787ec693450a3d1c5df5d6dc203' 2024-09-17T18:07:42.408 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T18:07:42.408 INFO:teuthology.packaging:ref: None 2024-09-17T18:07:42.408 INFO:teuthology.packaging:tag: None 2024-09-17T18:07:42.408 INFO:teuthology.packaging:branch: wip-bharath10-testing-2024-09-17-1348 2024-09-17T18:07:42.408 INFO:teuthology.packaging:sha1: 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:07:42.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-bharath10-testing-2024-09-17-1348 2024-09-17T18:07:42.546 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4975-g1889a165-1jammy 2024-09-17T18:07:42.548 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T18:07:42.555 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T18:07:42.555 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T18:07:42.561 INFO:teuthology.task.internal:Saving configuration 2024-09-17T18:07:42.569 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T18:07:42.576 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T18:07:42.600 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909236', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 18:02:12.684204', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEo9RhL9wLdluT93mUW6p6knx/kVAR+UnO2sDSz+2tEr'} 2024-09-17T18:07:42.600 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T18:07:42.606 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-17T18:07:42.606 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T18:07:42.642 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faf90097be0>, signals=[15]) 2024-09-17T18:07:42.642 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T18:07:42.648 INFO:teuthology.task.internal:Opening connections... 2024-09-17T18:07:42.649 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-09-17T18:07:42.650 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:07:42.734 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T18:07:42.740 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-09-17T18:07:42.833 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-09-17T18:07:42.833 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-09-17T18:07:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-17T18:07:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Ubuntu" 2024-09-17T18:07:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="22.04" 2024-09-17T18:07:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-17T18:07:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_CODENAME=jammy 2024-09-17T18:07:42.879 INFO:teuthology.orchestra.run.smithi105.stdout:ID=ubuntu 2024-09-17T18:07:42.880 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE=debian 2024-09-17T18:07:42.880 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-17T18:07:42.880 INFO:teuthology.orchestra.run.smithi105.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-17T18:07:42.880 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-17T18:07:42.880 INFO:teuthology.orchestra.run.smithi105.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-17T18:07:42.880 INFO:teuthology.orchestra.run.smithi105.stdout:UBUNTU_CODENAME=jammy 2024-09-17T18:07:42.880 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-09-17T18:07:42.907 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T18:07:42.916 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T18:07:42.921 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T18:07:42.921 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-09-17T18:07:42.926 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T18:07:42.934 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T18:07:42.934 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-09-17T18:07:42.973 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T18:07:43.003 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T18:07:43.032 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T18:07:43.032 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T18:07:43.033 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T18:07:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:07:43.037 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-09-17T18:07:43.083 INFO:teuthology.orchestra.run.smithi105.stdout:5.15.0-56-generic 2024-09-17T18:07:43.084 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.15.0-56-generic 2024-09-17T18:07:43.084 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get clean 2024-09-17T18:07:43.210 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2024-09-17T18:07:43.348 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-17T18:07:43.458 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T18:07:43.540 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-17T18:07:43.652 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-17T18:07:43.757 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-17T18:07:43.761 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-17T18:07:43.764 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-17T18:07:43.764 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-17T18:07:43.764 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-17T18:07:43.784 INFO:teuthology.orchestra.run.smithi105.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-17T18:07:43.786 INFO:teuthology.orchestra.run.smithi105.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-17T18:07:43.786 INFO:teuthology.orchestra.run.smithi105.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [628 kB] 2024-09-17T18:07:43.790 INFO:teuthology.orchestra.run.smithi105.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [903 kB] 2024-09-17T18:07:43.796 INFO:teuthology.orchestra.run.smithi105.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-17T18:07:43.797 INFO:teuthology.orchestra.run.smithi105.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-17T18:07:43.798 INFO:teuthology.orchestra.run.smithi105.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-17T18:07:43.798 INFO:teuthology.orchestra.run.smithi105.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-17T18:07:43.825 INFO:teuthology.orchestra.run.smithi105.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-17T18:07:43.825 INFO:teuthology.orchestra.run.smithi105.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-17T18:07:43.895 INFO:teuthology.orchestra.run.smithi105.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-17T18:07:44.033 INFO:teuthology.orchestra.run.smithi105.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [701 kB] 2024-09-17T18:07:44.227 INFO:teuthology.orchestra.run.smithi105.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,058 kB] 2024-09-17T18:07:44.388 INFO:teuthology.orchestra.run.smithi105.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [355 kB] 2024-09-17T18:07:44.402 INFO:teuthology.orchestra.run.smithi105.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-17T18:07:44.403 INFO:teuthology.orchestra.run.smithi105.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-17T18:07:44.404 INFO:teuthology.orchestra.run.smithi105.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,495 kB] 2024-09-17T18:07:44.477 INFO:teuthology.orchestra.run.smithi105.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [429 kB] 2024-09-17T18:07:44.492 INFO:teuthology.orchestra.run.smithi105.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-17T18:07:44.492 INFO:teuthology.orchestra.run.smithi105.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-17T18:07:44.511 INFO:teuthology.orchestra.run.smithi105.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-17T18:07:44.541 INFO:teuthology.orchestra.run.smithi105.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-17T18:07:44.547 INFO:teuthology.orchestra.run.smithi105.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-17T18:07:44.552 INFO:teuthology.orchestra.run.smithi105.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-17T18:07:44.552 INFO:teuthology.orchestra.run.smithi105.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-17T18:07:44.553 INFO:teuthology.orchestra.run.smithi105.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-17T18:07:44.630 INFO:teuthology.orchestra.run.smithi105.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-17T18:07:44.630 INFO:teuthology.orchestra.run.smithi105.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-17T18:07:44.632 INFO:teuthology.orchestra.run.smithi105.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-17T18:07:44.633 INFO:teuthology.orchestra.run.smithi105.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-17T18:07:44.634 INFO:teuthology.orchestra.run.smithi105.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-17T18:07:44.634 INFO:teuthology.orchestra.run.smithi105.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-17T18:07:44.634 INFO:teuthology.orchestra.run.smithi105.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-17T18:07:44.714 INFO:teuthology.orchestra.run.smithi105.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-17T18:07:44.715 INFO:teuthology.orchestra.run.smithi105.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-17T18:07:50.114 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 16.2 MB in 2s (7,218 kB/s) 2024-09-17T18:07:51.512 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2024-09-17T18:07:51.529 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T18:07:51.590 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2024-09-17T18:07:51.801 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2024-09-17T18:07:51.801 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2024-09-17T18:07:51.978 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T18:07:51.978 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T18:07:51.978 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T18:07:51.979 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T18:07:51.979 INFO:teuthology.orchestra.run.smithi105.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-17T18:07:51.979 INFO:teuthology.orchestra.run.smithi105.stdout: python2.7-minimal 2024-09-17T18:07:51.979 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T18:07:51.980 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2024-09-17T18:07:51.980 INFO:teuthology.orchestra.run.smithi105.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T18:07:51.980 INFO:teuthology.orchestra.run.smithi105.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-17T18:07:51.981 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-17T18:07:51.982 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2024-09-17T18:07:51.982 INFO:teuthology.orchestra.run.smithi105.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-17T18:07:51.982 INFO:teuthology.orchestra.run.smithi105.stdout:Recommended packages: 2024-09-17T18:07:51.982 INFO:teuthology.orchestra.run.smithi105.stdout: thermald 2024-09-17T18:07:52.035 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2024-09-17T18:07:52.035 INFO:teuthology.orchestra.run.smithi105.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-17T18:07:52.036 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-17T18:07:52.036 INFO:teuthology.orchestra.run.smithi105.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-17T18:07:52.038 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages will be upgraded: 2024-09-17T18:07:52.038 INFO:teuthology.orchestra.run.smithi105.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-17T18:07:52.215 INFO:teuthology.orchestra.run.smithi105.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T18:07:52.215 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 113 MB of archives. 2024-09-17T18:07:52.215 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-17T18:07:52.215 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-17T18:07:53.619 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-17T18:07:53.975 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-17T18:07:56.135 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-17T18:07:56.135 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-17T18:07:56.135 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-17T18:07:56.473 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-17T18:07:56.550 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-17T18:07:56.969 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 113 MB in 5s (25.1 MB/s) 2024-09-17T18:07:57.148 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-17T18:07:59.120 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-17T18:07:59.124 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T18:07:59.186 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:01.587 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-17T18:08:01.608 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T18:08:01.696 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:02.141 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-17T18:08:02.162 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T18:08:02.190 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:08.731 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T18:08:08.850 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T18:08:09.192 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T18:08:09.295 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T18:08:09.590 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-17T18:08:09.613 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-17T18:08:09.656 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T18:08:15.223 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-17T18:08:15.260 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-17T18:08:15.297 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:18.187 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-17T18:08:18.312 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-17T18:08:18.650 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-17T18:08:18.789 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:18.915 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-17T18:08:19.937 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:21.261 INFO:teuthology.orchestra.run.smithi105.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-17T18:08:21.261 INFO:teuthology.orchestra.run.smithi105.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-17T18:08:21.261 INFO:teuthology.orchestra.run.smithi105.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-17T18:08:21.262 INFO:teuthology.orchestra.run.smithi105.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-17T18:08:21.412 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:22.839 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-17T18:08:22.965 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-17T18:08:23.099 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:24.501 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-17T18:08:24.571 INFO:teuthology.orchestra.run.smithi105.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-17T18:08:24.571 INFO:teuthology.orchestra.run.smithi105.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-17T18:08:38.569 INFO:teuthology.orchestra.run.smithi105.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-17T18:08:38.569 INFO:teuthology.orchestra.run.smithi105.stdout:Sourcing file `/etc/default/grub' 2024-09-17T18:08:38.589 INFO:teuthology.orchestra.run.smithi105.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T18:08:38.616 INFO:teuthology.orchestra.run.smithi105.stdout:Generating grub configuration file ... 2024-09-17T18:08:38.872 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T18:08:38.886 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T18:08:39.157 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T18:08:39.161 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T18:08:39.230 INFO:teuthology.orchestra.run.smithi105.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T18:08:39.234 INFO:teuthology.orchestra.run.smithi105.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T18:08:39.384 INFO:teuthology.orchestra.run.smithi105.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T18:08:39.384 INFO:teuthology.orchestra.run.smithi105.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T18:08:39.384 INFO:teuthology.orchestra.run.smithi105.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T18:08:39.419 INFO:teuthology.orchestra.run.smithi105.stdout:done 2024-09-17T18:08:39.875 DEBUG:teuthology.orchestra.run.smithi105:> dpkg -s linux-image-generic 2024-09-17T18:08:39.901 INFO:teuthology.orchestra.run.smithi105.stdout:Package: linux-image-generic 2024-09-17T18:08:39.901 INFO:teuthology.orchestra.run.smithi105.stdout:Status: install ok installed 2024-09-17T18:08:39.901 INFO:teuthology.orchestra.run.smithi105.stdout:Priority: optional 2024-09-17T18:08:39.901 INFO:teuthology.orchestra.run.smithi105.stdout:Section: kernel 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Installed-Size: 21 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: amd64 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Source: linux-meta 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Version: 5.15.0.122.122 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T18:08:39.902 INFO:teuthology.orchestra.run.smithi105.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T18:08:39.903 INFO:teuthology.orchestra.run.smithi105.stdout:Recommends: thermald 2024-09-17T18:08:39.903 INFO:teuthology.orchestra.run.smithi105.stdout:Description: Generic Linux kernel image 2024-09-17T18:08:39.903 INFO:teuthology.orchestra.run.smithi105.stdout: This package will always depend on the latest generic kernel image 2024-09-17T18:08:39.903 INFO:teuthology.orchestra.run.smithi105.stdout: available. 2024-09-17T18:08:39.903 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-17T18:08:39.903 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T18:08:39.903 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T18:08:39.904 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T18:08:39.904 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T18:08:39.904 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get clean 2024-09-17T18:08:40.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2024-09-17T18:08:40.197 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T18:08:40.319 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T18:08:40.400 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T18:08:40.482 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T18:08:41.666 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2024-09-17T18:08:41.685 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-17T18:08:41.738 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2024-09-17T18:08:41.923 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2024-09-17T18:08:41.924 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2024-09-17T18:08:42.091 INFO:teuthology.orchestra.run.smithi105.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-17T18:08:42.091 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T18:08:42.092 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T18:08:42.092 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T18:08:42.092 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T18:08:42.092 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T18:08:42.092 INFO:teuthology.orchestra.run.smithi105.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T18:08:42.092 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T18:08:42.133 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-17T18:08:42.135 DEBUG:teuthology.orchestra.run.smithi105:> dpkg -s linux-image-generic 2024-09-17T18:08:42.193 INFO:teuthology.orchestra.run.smithi105.stdout:Package: linux-image-generic 2024-09-17T18:08:42.193 INFO:teuthology.orchestra.run.smithi105.stdout:Status: install ok installed 2024-09-17T18:08:42.193 INFO:teuthology.orchestra.run.smithi105.stdout:Priority: optional 2024-09-17T18:08:42.193 INFO:teuthology.orchestra.run.smithi105.stdout:Section: kernel 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Installed-Size: 21 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Maintainer: Ubuntu Kernel Team 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: amd64 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Source: linux-meta 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Version: 5.15.0.122.122 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Recommends: thermald 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout:Description: Generic Linux kernel image 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout: This package will always depend on the latest generic kernel image 2024-09-17T18:08:42.194 INFO:teuthology.orchestra.run.smithi105.stdout: available. 2024-09-17T18:08:42.195 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-09-17T18:08:42.238 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.FTAd88LCIU 2024-09-17T18:08:42.239 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub/grub.cfg /tmp/tmp.FTAd88LCIU 2024-09-17T18:08:42.294 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.FTAd88LCIU 2024-09-17T18:08:42.415 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.FTAd88LCIU is 10KB 2024-09-17T18:08:42.428 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.FTAd88LCIU 2024-09-17T18:08:42.434 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-17T18:08:42.490 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:08:42.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-17T18:08:42.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-17T18:08:42.557 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-17T18:08:42.557 DEBUG:teuthology.orchestra.run.smithi105:> sudo update-grub 2024-09-17T18:08:43.836 INFO:teuthology.orchestra.run.smithi105.stderr:Sourcing file `/etc/default/grub' 2024-09-17T18:08:43.837 INFO:teuthology.orchestra.run.smithi105.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-17T18:08:43.840 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-09-17T18:08:44.030 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-17T18:08:44.036 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-17T18:08:44.267 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-17T18:08:44.272 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-17T18:08:44.308 INFO:teuthology.orchestra.run.smithi105.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-17T18:08:44.313 INFO:teuthology.orchestra.run.smithi105.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-17T18:08:44.422 INFO:teuthology.orchestra.run.smithi105.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-17T18:08:44.422 INFO:teuthology.orchestra.run.smithi105.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-17T18:08:44.422 INFO:teuthology.orchestra.run.smithi105.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-17T18:08:44.444 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-09-17T18:08:44.460 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-09-17T18:09:14.489 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T18:09:14.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:09:14.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:09:32.989 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-09-17T18:09:41.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:09:41.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:09:45.053 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-09-17T18:09:57.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:09:57.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:10:15.485 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-09-17T18:10:30.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:10:30.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:10:31.162 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-09-17T18:10:32.024 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:10:32.024 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-17T18:10:32.024 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-09-17T18:10:32.069 INFO:teuthology.orchestra.run.smithi105.stdout:5.15.0-122-generic 2024-09-17T18:10:32.070 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-17T18:10:32.070 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T18:10:32.070 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-17T18:10:33.070 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T18:10:33.071 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T18:10:33.170 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-09-17T18:10:33.195 DEBUG:teuthology.parallel:result is None 2024-09-17T18:10:33.195 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T18:10:33.204 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T18:10:33.204 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T18:10:33.209 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T18:10:33.254 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T18:10:33.279 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T18:10:33.279 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T18:10:33.300 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T18:10:33.366 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T18:10:33.366 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T18:10:33.394 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:10:33.402 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:10:33.403 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T18:10:33.410 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T18:10:33.410 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T18:10:33.457 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T18:10:33.466 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T18:10:33.466 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T18:10:33.506 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T18:10:33.554 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T18:10:33.607 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:10:33.607 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T18:10:33.668 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-09-17T18:10:33.744 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T18:10:33.751 INFO:teuthology.task.internal:Starting timer... 2024-09-17T18:10:33.751 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T18:10:33.761 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T18:10:33.769 DEBUG:teuthology.task.selinux:Excluding smithi105: OS 'ubuntu' does not support SELinux 2024-09-17T18:10:33.770 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T18:10:33.770 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-17T18:10:33.770 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T18:10:33.770 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T18:10:33.778 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-17T18:10:33.781 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-17T18:10:33.900 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T18:10:33.912 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T18:10:33.913 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T18:18:07.475 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2024-09-17T18:18:07.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:18:07.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T18:18:07.572 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-09-17T18:18:07.649 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-09-17T18:18:07.649 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T18:18:08.788 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T18:18:08.789 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T18:18:08.789 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T18:18:08.822 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-17T18:18:08.822 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Command line: ntpd -gq 2024-09-17T18:18:08.822 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: ---------------------------------------------------- 2024-09-17T18:18:08.822 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: ntp-4 is maintained by Network Time Foundation, 2024-09-17T18:18:08.823 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-17T18:18:08.823 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: corporation. Support and training for ntp-4 are 2024-09-17T18:18:08.823 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: available at https://www.nwtime.org/support 2024-09-17T18:18:08.823 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: ---------------------------------------------------- 2024-09-17T18:18:08.823 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: proto: precision = 0.045 usec (-24) 2024-09-17T18:18:08.824 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: basedate set to 2022-02-04 2024-09-17T18:18:08.824 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: gps base set to 2022-02-06 (week 2196) 2024-09-17T18:18:08.824 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T18:18:08.824 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: restrict ::: KOD does nothing without LIMITED. 2024-09-17T18:18:08.824 INFO:teuthology.orchestra.run.smithi105.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-17T18:18:08.824 INFO:teuthology.orchestra.run.smithi105.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-17T18:18:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listen and drop on 0 v6wildcard [::]:123 2024-09-17T18:18:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-17T18:18:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listen normally on 2 lo 127.0.0.1:123 2024-09-17T18:18:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listen normally on 3 enp3s0f1 172.21.15.105:123 2024-09-17T18:18:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listen normally on 4 lo [::1]:123 2024-09-17T18:18:08.825 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d429%5]:123 2024-09-17T18:18:08.826 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:08 ntpd[18507]: Listening on routing socket on fd #22 for interface updates 2024-09-17T18:18:09.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:09 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:09.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:09 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:09.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:09 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:09.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:09 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:09.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:09 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:09.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:09 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:10.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:10.825 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:10 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:11.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:11 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:11.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:11 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:11.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:11 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:11.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:11 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:11.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:11 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:11.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:11 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:12.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:12 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:13.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:13.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:13.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:13.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:13.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:13.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:13.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:13.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:13 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:14.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:14.825 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:14 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:15.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:15.823 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:15.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-17T18:18:15.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: can't open /var/log/ntpstats/rawstats.20240917: Permission denied 2024-09-17T18:18:15.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-17T18:18:15.824 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: can't open /var/log/ntpstats/peerstats.20240917: Permission denied 2024-09-17T18:18:15.824 INFO:teuthology.orchestra.run.smithi105.stdout:17 Sep 18:18:15 ntpd[18507]: ntpd: time slew -0.000438 s 2024-09-17T18:18:15.824 INFO:teuthology.orchestra.run.smithi105.stdout:ntpd: time slew -0.000438s 2024-09-17T18:18:15.825 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-17T18:18:15.825 INFO:teuthology.orchestra.run.smithi105.stderr:17 Sep 18:18:15 ntpd[18507]: can't open /var/log/ntpstats/loopstats.20240917: Permission denied 2024-09-17T18:18:15.870 INFO:teuthology.orchestra.run.smithi105.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T18:18:15.871 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================== 2024-09-17T18:18:15.871 INFO:teuthology.orchestra.run.smithi105.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T18:18:15.871 INFO:teuthology.orchestra.run.smithi105.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T18:18:15.871 INFO:teuthology.orchestra.run.smithi105.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T18:18:15.871 INFO:teuthology.orchestra.run.smithi105.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-17T18:18:15.872 INFO:teuthology.run_tasks:Running task install... 2024-09-17T18:18:15.880 DEBUG:teuthology.task.install:project ceph 2024-09-17T18:18:15.881 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'}} 2024-09-17T18:18:15.881 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1889a165b5128787ec693450a3d1c5df5d6dc203'} 2024-09-17T18:18:15.881 INFO:teuthology.task.install:Using flavor: default 2024-09-17T18:18:15.887 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T18:18:15.887 INFO:teuthology.task.install:extra packages: [] 2024-09-17T18:18:15.887 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-key list | grep Ceph 2024-09-17T18:18:15.972 INFO:teuthology.orchestra.run.smithi105.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-17T18:18:15.998 INFO:teuthology.orchestra.run.smithi105.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-17T18:18:15.998 INFO:teuthology.orchestra.run.smithi105.stdout:uid [ unknown] Ceph.com (release key) 2024-09-17T18:18:15.999 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-17T18:18:15.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:18:16.152 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default/ 2024-09-17T18:18:16.152 INFO:teuthology.task.install.deb:Package version is 19.3.0-4975-g1889a165-1jammy 2024-09-17T18:18:16.267 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:18:16.267 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-17T18:18:16.281 DEBUG:teuthology.orchestra.run.smithi105:> sudo apt-get update 2024-09-17T18:18:16.449 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-17T18:18:16.530 INFO:teuthology.orchestra.run.smithi105.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy InRelease 2024-09-17T18:18:16.570 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-17T18:18:16.581 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-17T18:18:16.632 INFO:teuthology.orchestra.run.smithi105.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-17T18:18:16.651 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-17T18:18:16.682 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-09-17T18:18:16.733 INFO:teuthology.orchestra.run.smithi105.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-17T18:18:16.739 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-17T18:18:16.870 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 28.6 kB in 0s (69.9 kB/s) 2024-09-17T18:18:17.907 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2024-09-17T18:18:17.924 DEBUG:teuthology.orchestra.run.smithi105:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4975-g1889a165-1jammy cephadm=19.3.0-4975-g1889a165-1jammy ceph-mds=19.3.0-4975-g1889a165-1jammy ceph-mgr=19.3.0-4975-g1889a165-1jammy ceph-common=19.3.0-4975-g1889a165-1jammy ceph-fuse=19.3.0-4975-g1889a165-1jammy ceph-test=19.3.0-4975-g1889a165-1jammy ceph-volume=19.3.0-4975-g1889a165-1jammy radosgw=19.3.0-4975-g1889a165-1jammy python3-rados=19.3.0-4975-g1889a165-1jammy python3-rgw=19.3.0-4975-g1889a165-1jammy python3-cephfs=19.3.0-4975-g1889a165-1jammy python3-rbd=19.3.0-4975-g1889a165-1jammy libcephfs2=19.3.0-4975-g1889a165-1jammy libcephfs-dev=19.3.0-4975-g1889a165-1jammy librados2=19.3.0-4975-g1889a165-1jammy librbd1=19.3.0-4975-g1889a165-1jammy rbd-fuse=19.3.0-4975-g1889a165-1jammy 2024-09-17T18:18:17.987 INFO:teuthology.orchestra.run.smithi105.stdout:Reading package lists... 2024-09-17T18:18:18.198 INFO:teuthology.orchestra.run.smithi105.stdout:Building dependency tree... 2024-09-17T18:18:18.199 INFO:teuthology.orchestra.run.smithi105.stdout:Reading state information... 2024-09-17T18:18:18.408 INFO:teuthology.orchestra.run.smithi105.stdout:The following packages were automatically installed and are no longer required: 2024-09-17T18:18:18.408 INFO:teuthology.orchestra.run.smithi105.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-17T18:18:18.408 INFO:teuthology.orchestra.run.smithi105.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-17T18:18:18.408 INFO:teuthology.orchestra.run.smithi105.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-17T18:18:18.409 INFO:teuthology.orchestra.run.smithi105.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-17T18:18:18.409 INFO:teuthology.orchestra.run.smithi105.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-17T18:18:18.409 INFO:teuthology.orchestra.run.smithi105.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-17T18:18:18.410 INFO:teuthology.orchestra.run.smithi105.stdout:The following additional packages will be installed: 2024-09-17T18:18:18.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-17T18:18:18.410 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-17T18:18:18.410 INFO:teuthology.orchestra.run.smithi105.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-17T18:18:18.410 INFO:teuthology.orchestra.run.smithi105.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-17T18:18:18.410 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-17T18:18:18.411 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-17T18:18:18.412 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet zip 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout:Suggested packages: 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout:Recommended packages: 2024-09-17T18:18:18.413 INFO:teuthology.orchestra.run.smithi105.stdout: btrfs-tools 2024-09-17T18:18:18.516 INFO:teuthology.orchestra.run.smithi105.stdout:The following NEW packages will be installed: 2024-09-17T18:18:18.516 INFO:teuthology.orchestra.run.smithi105.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-17T18:18:18.516 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-17T18:18:18.517 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-17T18:18:18.517 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-17T18:18:18.517 INFO:teuthology.orchestra.run.smithi105.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-17T18:18:18.517 INFO:teuthology.orchestra.run.smithi105.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-17T18:18:18.518 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-17T18:18:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-17T18:18:18.581 INFO:teuthology.orchestra.run.smithi105.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-17T18:18:18.581 INFO:teuthology.orchestra.run.smithi105.stdout:Need to get 244 MB of archives. 2024-09-17T18:18:18.581 INFO:teuthology.orchestra.run.smithi105.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-17T18:18:18.581 INFO:teuthology.orchestra.run.smithi105.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-17T18:18:18.662 INFO:teuthology.orchestra.run.smithi105.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4975-g1889a165-1jammy [3,624 kB] 2024-09-17T18:18:18.669 INFO:teuthology.orchestra.run.smithi105.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-17T18:18:18.672 INFO:teuthology.orchestra.run.smithi105.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-17T18:18:18.689 INFO:teuthology.orchestra.run.smithi105.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-17T18:18:18.749 INFO:teuthology.orchestra.run.smithi105.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-17T18:18:18.754 INFO:teuthology.orchestra.run.smithi105.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-17T18:18:18.766 INFO:teuthology.orchestra.run.smithi105.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-17T18:18:18.770 INFO:teuthology.orchestra.run.smithi105.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-17T18:18:18.771 INFO:teuthology.orchestra.run.smithi105.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-17T18:18:18.772 INFO:teuthology.orchestra.run.smithi105.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-17T18:18:18.773 INFO:teuthology.orchestra.run.smithi105.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-17T18:18:18.781 INFO:teuthology.orchestra.run.smithi105.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-17T18:18:18.783 INFO:teuthology.orchestra.run.smithi105.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-17T18:18:18.786 INFO:teuthology.orchestra.run.smithi105.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-17T18:18:18.787 INFO:teuthology.orchestra.run.smithi105.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-17T18:18:18.787 INFO:teuthology.orchestra.run.smithi105.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-17T18:18:18.805 INFO:teuthology.orchestra.run.smithi105.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-17T18:18:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-17T18:18:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-17T18:18:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-17T18:18:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-17T18:18:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-17T18:18:18.811 INFO:teuthology.orchestra.run.smithi105.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-17T18:18:18.822 INFO:teuthology.orchestra.run.smithi105.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-17T18:18:18.822 INFO:teuthology.orchestra.run.smithi105.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-17T18:18:18.823 INFO:teuthology.orchestra.run.smithi105.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-17T18:18:18.839 INFO:teuthology.orchestra.run.smithi105.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-17T18:18:18.839 INFO:teuthology.orchestra.run.smithi105.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-17T18:18:18.842 INFO:teuthology.orchestra.run.smithi105.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-17T18:18:18.842 INFO:teuthology.orchestra.run.smithi105.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-17T18:18:18.843 INFO:teuthology.orchestra.run.smithi105.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-17T18:18:18.843 INFO:teuthology.orchestra.run.smithi105.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-17T18:18:18.844 INFO:teuthology.orchestra.run.smithi105.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-17T18:18:18.847 INFO:teuthology.orchestra.run.smithi105.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4975-g1889a165-1jammy [3,332 kB] 2024-09-17T18:18:18.856 INFO:teuthology.orchestra.run.smithi105.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-17T18:18:18.856 INFO:teuthology.orchestra.run.smithi105.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-17T18:18:18.861 INFO:teuthology.orchestra.run.smithi105.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-17T18:18:18.873 INFO:teuthology.orchestra.run.smithi105.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-17T18:18:18.874 INFO:teuthology.orchestra.run.smithi105.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-17T18:18:18.879 INFO:teuthology.orchestra.run.smithi105.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-17T18:18:18.880 INFO:teuthology.orchestra.run.smithi105.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-17T18:18:18.880 INFO:teuthology.orchestra.run.smithi105.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-17T18:18:18.881 INFO:teuthology.orchestra.run.smithi105.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-17T18:18:18.882 INFO:teuthology.orchestra.run.smithi105.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-17T18:18:18.890 INFO:teuthology.orchestra.run.smithi105.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-17T18:18:18.891 INFO:teuthology.orchestra.run.smithi105.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-17T18:18:18.893 INFO:teuthology.orchestra.run.smithi105.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-17T18:18:18.893 INFO:teuthology.orchestra.run.smithi105.stdout:Get:49 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4975-g1889a165-1jammy [722 kB] 2024-09-17T18:18:18.904 INFO:teuthology.orchestra.run.smithi105.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4975-g1889a165-1jammy [356 kB] 2024-09-17T18:18:18.908 INFO:teuthology.orchestra.run.smithi105.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-17T18:18:18.910 INFO:teuthology.orchestra.run.smithi105.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4975-g1889a165-1jammy [32.8 kB] 2024-09-17T18:18:18.910 INFO:teuthology.orchestra.run.smithi105.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-17T18:18:18.911 INFO:teuthology.orchestra.run.smithi105.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-17T18:18:18.911 INFO:teuthology.orchestra.run.smithi105.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4975-g1889a165-1jammy [180 kB] 2024-09-17T18:18:18.912 INFO:teuthology.orchestra.run.smithi105.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-17T18:18:18.913 INFO:teuthology.orchestra.run.smithi105.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4975-g1889a165-1jammy [71.3 kB] 2024-09-17T18:18:18.914 INFO:teuthology.orchestra.run.smithi105.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-17T18:18:18.915 INFO:teuthology.orchestra.run.smithi105.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4975-g1889a165-1jammy [338 kB] 2024-09-17T18:18:18.920 INFO:teuthology.orchestra.run.smithi105.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4975-g1889a165-1jammy [7,966 kB] 2024-09-17T18:18:18.967 INFO:teuthology.orchestra.run.smithi105.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-17T18:18:18.968 INFO:teuthology.orchestra.run.smithi105.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-17T18:18:18.970 INFO:teuthology.orchestra.run.smithi105.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-17T18:18:18.970 INFO:teuthology.orchestra.run.smithi105.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-17T18:18:18.971 INFO:teuthology.orchestra.run.smithi105.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-17T18:18:18.988 INFO:teuthology.orchestra.run.smithi105.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-17T18:18:18.990 INFO:teuthology.orchestra.run.smithi105.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-17T18:18:18.990 INFO:teuthology.orchestra.run.smithi105.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-17T18:18:19.005 INFO:teuthology.orchestra.run.smithi105.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-17T18:18:19.005 INFO:teuthology.orchestra.run.smithi105.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-17T18:18:19.016 INFO:teuthology.orchestra.run.smithi105.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-17T18:18:19.017 INFO:teuthology.orchestra.run.smithi105.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-17T18:18:19.017 INFO:teuthology.orchestra.run.smithi105.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-17T18:18:19.017 INFO:teuthology.orchestra.run.smithi105.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-17T18:18:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-17T18:18:19.018 INFO:teuthology.orchestra.run.smithi105.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-17T18:18:19.022 INFO:teuthology.orchestra.run.smithi105.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-17T18:18:19.034 INFO:teuthology.orchestra.run.smithi105.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-17T18:18:19.035 INFO:teuthology.orchestra.run.smithi105.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-17T18:18:19.036 INFO:teuthology.orchestra.run.smithi105.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-17T18:18:19.050 INFO:teuthology.orchestra.run.smithi105.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-17T18:18:19.052 INFO:teuthology.orchestra.run.smithi105.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4975-g1889a165-1jammy [112 kB] 2024-09-17T18:18:19.053 INFO:teuthology.orchestra.run.smithi105.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-17T18:18:19.054 INFO:teuthology.orchestra.run.smithi105.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-17T18:18:19.054 INFO:teuthology.orchestra.run.smithi105.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4975-g1889a165-1jammy [473 kB] 2024-09-17T18:18:19.054 INFO:teuthology.orchestra.run.smithi105.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-17T18:18:19.058 INFO:teuthology.orchestra.run.smithi105.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-17T18:18:19.059 INFO:teuthology.orchestra.run.smithi105.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-17T18:18:19.092 INFO:teuthology.orchestra.run.smithi105.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-17T18:18:19.093 INFO:teuthology.orchestra.run.smithi105.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4975-g1889a165-1jammy [28.2 MB] 2024-09-17T18:18:19.093 INFO:teuthology.orchestra.run.smithi105.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-17T18:18:19.094 INFO:teuthology.orchestra.run.smithi105.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-17T18:18:19.095 INFO:teuthology.orchestra.run.smithi105.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-17T18:18:19.095 INFO:teuthology.orchestra.run.smithi105.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-17T18:18:19.096 INFO:teuthology.orchestra.run.smithi105.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-17T18:18:19.104 INFO:teuthology.orchestra.run.smithi105.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-17T18:18:19.104 INFO:teuthology.orchestra.run.smithi105.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-17T18:18:19.104 INFO:teuthology.orchestra.run.smithi105.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-17T18:18:19.110 INFO:teuthology.orchestra.run.smithi105.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-17T18:18:19.111 INFO:teuthology.orchestra.run.smithi105.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-17T18:18:19.658 INFO:teuthology.orchestra.run.smithi105.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4975-g1889a165-1jammy [5,209 kB] 2024-09-17T18:18:19.724 INFO:teuthology.orchestra.run.smithi105.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4975-g1889a165-1jammy [251 kB] 2024-09-17T18:18:19.727 INFO:teuthology.orchestra.run.smithi105.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4975-g1889a165-1jammy [124 kB] 2024-09-17T18:18:19.728 INFO:teuthology.orchestra.run.smithi105.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4975-g1889a165-1jammy [1,561 kB] 2024-09-17T18:18:19.741 INFO:teuthology.orchestra.run.smithi105.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4975-g1889a165-1jammy [6,310 kB] 2024-09-17T18:18:19.863 INFO:teuthology.orchestra.run.smithi105.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4975-g1889a165-1jammy [23.1 MB] 2024-09-17T18:18:20.431 INFO:teuthology.orchestra.run.smithi105.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4975-g1889a165-1jammy [14.0 kB] 2024-09-17T18:18:20.431 INFO:teuthology.orchestra.run.smithi105.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4975-g1889a165-1jammy [880 kB] 2024-09-17T18:18:20.447 INFO:teuthology.orchestra.run.smithi105.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4975-g1889a165-1jammy [2,452 kB] 2024-09-17T18:18:20.487 INFO:teuthology.orchestra.run.smithi105.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4975-g1889a165-1jammy [813 kB] 2024-09-17T18:18:20.497 INFO:teuthology.orchestra.run.smithi105.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4975-g1889a165-1jammy [160 kB] 2024-09-17T18:18:20.499 INFO:teuthology.orchestra.run.smithi105.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4975-g1889a165-1jammy [61.5 MB] 2024-09-17T18:18:22.385 INFO:teuthology.orchestra.run.smithi105.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4975-g1889a165-1jammy [8,625 kB] 2024-09-17T18:18:22.554 INFO:teuthology.orchestra.run.smithi105.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4975-g1889a165-1jammy [14.2 kB] 2024-09-17T18:18:22.554 INFO:teuthology.orchestra.run.smithi105.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4975-g1889a165-1jammy [48.5 MB] 2024-09-17T18:18:24.168 INFO:teuthology.orchestra.run.smithi105.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4975-g1889a165-1jammy [130 kB] 2024-09-17T18:18:24.170 INFO:teuthology.orchestra.run.smithi105.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4975-g1889a165-1jammy [39.0 kB] 2024-09-17T18:18:24.172 INFO:teuthology.orchestra.run.smithi105.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4975-g1889a165-1jammy [15.8 MB] 2024-09-17T18:18:24.634 INFO:teuthology.orchestra.run.smithi105.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-bharath10-testing-2024-09-17-1348/1889a165b5128787ec693450a3d1c5df5d6dc203/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4975-g1889a165-1jammy [91.3 kB] 2024-09-17T18:18:25.218 INFO:teuthology.orchestra.run.smithi105.stdout:Fetched 244 MB in 6s (39.9 MB/s) 2024-09-17T18:18:25.353 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-17T18:18:25.451 INFO:teuthology.orchestra.run.smithi105.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-17T18:18:25.457 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-17T18:18:25.527 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T18:18:25.847 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-17T18:18:25.873 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-17T18:18:25.913 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T18:18:26.250 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-17T18:18:26.276 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-17T18:18:26.316 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T18:18:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-17T18:18:26.772 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T18:18:26.820 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T18:18:27.207 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-17T18:18:27.233 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T18:18:27.273 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T18:18:27.601 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-17T18:18:27.628 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-17T18:18:27.668 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T18:18:28.013 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-17T18:18:28.039 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-17T18:18:28.079 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T18:18:28.340 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librados2. 2024-09-17T18:18:28.367 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../007-librados2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:28.407 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librados2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:28.885 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libnbd0. 2024-09-17T18:18:28.911 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-17T18:18:28.951 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-17T18:18:29.213 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librbd1. 2024-09-17T18:18:29.238 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../009-librbd1_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:29.271 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librbd1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:29.707 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libcephfs2. 2024-09-17T18:18:29.733 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:29.773 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libcephfs2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:30.059 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rados. 2024-09-17T18:18:30.085 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:30.126 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rados (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:30.445 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-17T18:18:30.471 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:18:30.512 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-ceph-argparse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:30.731 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cephfs. 2024-09-17T18:18:30.746 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:30.781 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cephfs (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:31.075 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-17T18:18:31.101 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:18:31.142 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:31.470 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-17T18:18:31.496 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-17T18:18:31.537 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T18:18:31.873 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-prettytable. 2024-09-17T18:18:31.899 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-17T18:18:31.939 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-17T18:18:32.184 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rbd. 2024-09-17T18:18:32.199 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:32.233 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rbd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:32.546 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-17T18:18:32.571 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-17T18:18:32.620 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T18:18:32.898 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-17T18:18:32.913 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-17T18:18:32.956 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T18:18:33.242 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-17T18:18:33.268 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-17T18:18:33.308 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T18:18:33.587 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package lua5.1. 2024-09-17T18:18:33.601 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-17T18:18:33.645 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-17T18:18:33.940 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package lua-any. 2024-09-17T18:18:33.966 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-17T18:18:34.006 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-17T18:18:34.242 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package zip. 2024-09-17T18:18:34.257 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-17T18:18:34.300 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking zip (3.0-12build2) ... 2024-09-17T18:18:34.587 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package unzip. 2024-09-17T18:18:34.613 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-17T18:18:34.645 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-17T18:18:35.024 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package luarocks. 2024-09-17T18:18:35.050 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-17T18:18:35.090 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-17T18:18:35.473 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package librgw2. 2024-09-17T18:18:35.500 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../026-librgw2_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:35.565 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking librgw2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:36.418 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rgw. 2024-09-17T18:18:36.444 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:36.484 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:36.804 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-17T18:18:36.830 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-17T18:18:36.870 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T18:18:37.140 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libradosstriper1. 2024-09-17T18:18:37.166 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:37.206 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libradosstriper1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:37.627 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-common. 2024-09-17T18:18:37.653 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:37.693 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:39.098 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-base. 2024-09-17T18:18:39.125 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:39.166 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-base (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:39.694 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-17T18:18:39.721 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-17T18:18:39.760 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-17T18:18:40.032 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cheroot. 2024-09-17T18:18:40.048 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-17T18:18:40.090 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T18:18:40.377 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-17T18:18:40.403 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-17T18:18:40.443 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-17T18:18:40.746 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-17T18:18:40.773 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-17T18:18:40.812 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-17T18:18:41.124 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-17T18:18:41.150 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-17T18:18:41.190 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-17T18:18:41.468 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-tempora. 2024-09-17T18:18:41.484 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-17T18:18:41.526 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-17T18:18:41.813 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-portend. 2024-09-17T18:18:41.839 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-17T18:18:41.879 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-17T18:18:42.190 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-17T18:18:42.217 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-17T18:18:42.257 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-17T18:18:42.560 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-17T18:18:42.586 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-17T18:18:42.626 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-17T18:18:42.980 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-natsort. 2024-09-17T18:18:43.006 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-17T18:18:43.046 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-17T18:18:43.374 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-logutils. 2024-09-17T18:18:43.401 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-17T18:18:43.441 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-17T18:18:43.694 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-17T18:18:43.720 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-17T18:18:43.752 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-17T18:18:44.055 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-mako. 2024-09-17T18:18:44.081 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-17T18:18:44.121 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T18:18:44.467 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-17T18:18:44.493 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-17T18:18:44.533 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-17T18:18:44.836 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-17T18:18:44.862 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-17T18:18:44.902 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-17T18:18:45.188 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-webob. 2024-09-17T18:18:45.214 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-17T18:18:45.255 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T18:18:45.566 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-waitress. 2024-09-17T18:18:45.593 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-17T18:18:45.627 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T18:18:45.919 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-tempita. 2024-09-17T18:18:45.945 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-17T18:18:45.986 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T18:18:46.288 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-paste. 2024-09-17T18:18:46.315 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-17T18:18:46.355 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T18:18:46.850 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-17T18:18:46.865 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-17T18:18:46.908 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T18:18:47.194 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-17T18:18:47.220 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-17T18:18:47.261 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-17T18:18:47.564 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-webtest. 2024-09-17T18:18:47.590 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-17T18:18:47.630 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-17T18:18:47.917 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pecan. 2024-09-17T18:18:47.932 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-17T18:18:47.966 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T18:18:48.296 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-certifi. 2024-09-17T18:18:48.322 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-17T18:18:48.362 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-17T18:18:48.691 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-idna. 2024-09-17T18:18:48.718 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-17T18:18:48.757 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T18:18:49.043 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-urllib3. 2024-09-17T18:18:49.059 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-17T18:18:49.102 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T18:18:49.430 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-requests. 2024-09-17T18:18:49.456 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-17T18:18:49.496 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T18:18:49.808 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-17T18:18:49.834 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-17T18:18:49.876 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T18:18:50.194 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-17T18:18:50.209 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:18:50.244 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:50.615 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-17T18:18:50.642 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:50.682 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:50.993 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-17T18:18:51.019 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-17T18:18:51.059 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-17T18:18:51.337 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-openssl. 2024-09-17T18:18:51.364 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-17T18:18:51.395 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-17T18:18:51.715 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr. 2024-09-17T18:18:51.742 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:52.107 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:52.927 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mon. 2024-09-17T18:18:52.953 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:52.993 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mon (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:53.564 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-osd. 2024-09-17T18:18:53.590 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:53.630 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-osd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:54.675 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph. 2024-09-17T18:18:54.690 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../067-ceph_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:54.733 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:55.052 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-fuse. 2024-09-17T18:18:55.079 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:55.119 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:55.489 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mds. 2024-09-17T18:18:55.515 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:55.555 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mds (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:55.908 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package cephadm. 2024-09-17T18:18:55.923 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../070-cephadm_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:18:55.966 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:56.269 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-17T18:18:56.296 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-17T18:18:56.335 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-17T18:18:56.822 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-17T18:18:56.849 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T18:18:56.888 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T18:18:57.927 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-babel. 2024-09-17T18:18:57.943 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-17T18:18:57.976 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T18:18:58.263 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-jinja2. 2024-09-17T18:18:58.290 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-17T18:18:58.321 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T18:18:58.641 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-17T18:18:58.656 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:18:58.699 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:18:59.002 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-17T18:18:59.030 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-17T18:18:59.061 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-17T18:18:59.347 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-routes. 2024-09-17T18:18:59.373 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-17T18:18:59.413 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T18:18:59.725 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-17T18:18:59.752 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:18:59.791 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:02.630 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-17T18:19:02.660 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-17T18:19:02.704 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T18:19:03.177 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-joblib. 2024-09-17T18:19:03.202 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-17T18:19:03.249 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T18:19:03.603 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-17T18:19:03.630 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-17T18:19:03.669 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-17T18:19:03.972 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-sklearn. 2024-09-17T18:19:03.999 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-17T18:19:04.124 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T18:19:04.858 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-17T18:19:04.886 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:19:04.925 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:05.853 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-cachetools. 2024-09-17T18:19:05.880 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-17T18:19:05.911 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-17T18:19:06.197 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-17T18:19:06.225 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-17T18:19:06.264 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-17T18:19:06.592 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-17T18:19:06.620 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-17T18:19:06.659 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T18:19:06.953 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-rsa. 2024-09-17T18:19:06.981 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-17T18:19:07.019 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-17T18:19:07.549 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-google-auth. 2024-09-17T18:19:07.576 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-17T18:19:07.615 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-17T18:19:07.918 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-17T18:19:07.934 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-17T18:19:07.968 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T18:19:08.238 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-websocket. 2024-09-17T18:19:08.266 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-17T18:19:08.304 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-17T18:19:08.641 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-17T18:19:08.657 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-17T18:19:08.699 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T18:19:09.330 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-17T18:19:09.359 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:19:09.396 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:09.716 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-17T18:19:09.743 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-17T18:19:09.782 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T18:19:10.069 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-17T18:19:10.097 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T18:19:10.127 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T18:19:10.388 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package jq. 2024-09-17T18:19:10.404 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-17T18:19:10.446 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-17T18:19:10.690 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package socat. 2024-09-17T18:19:10.718 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-17T18:19:10.757 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-17T18:19:11.060 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package xmlstarlet. 2024-09-17T18:19:11.077 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-17T18:19:11.118 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-17T18:19:11.473 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-test. 2024-09-17T18:19:11.489 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:19:11.530 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-test (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:13.660 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package ceph-volume. 2024-09-17T18:19:13.689 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4975-g1889a165-1jammy_all.deb ... 2024-09-17T18:19:13.727 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking ceph-volume (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:13.997 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-17T18:19:14.013 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:19:14.047 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking libcephfs-dev (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:14.325 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-17T18:19:14.353 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-17T18:19:14.391 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T18:19:14.704 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-17T18:19:14.732 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-17T18:19:14.770 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-17T18:19:15.124 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package nvme-cli. 2024-09-17T18:19:15.153 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-17T18:19:15.191 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T18:19:15.605 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package pkg-config. 2024-09-17T18:19:15.633 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-17T18:19:15.662 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T18:19:15.924 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-17T18:19:15.940 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-17T18:19:15.982 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-17T18:19:16.394 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-17T18:19:16.422 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-17T18:19:16.460 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-17T18:19:16.738 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pastescript. 2024-09-17T18:19:16.766 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-17T18:19:16.804 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-17T18:19:17.116 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pluggy. 2024-09-17T18:19:17.144 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-17T18:19:17.182 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-17T18:19:17.502 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-psutil. 2024-09-17T18:19:17.530 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-17T18:19:17.568 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-17T18:19:18.196 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-py. 2024-09-17T18:19:18.224 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-17T18:19:18.263 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-17T18:19:18.599 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pygments. 2024-09-17T18:19:18.628 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-17T18:19:18.666 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T18:19:19.103 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-17T18:19:19.132 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-17T18:19:19.169 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-17T18:19:19.464 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-toml. 2024-09-17T18:19:19.492 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-17T18:19:19.530 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-17T18:19:19.808 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-pytest. 2024-09-17T18:19:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-17T18:19:19.899 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T18:19:20.255 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package python3-simplejson. 2024-09-17T18:19:20.283 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-17T18:19:20.321 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-17T18:19:20.599 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-17T18:19:20.615 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-17T18:19:20.657 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-17T18:19:21.320 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package radosgw. 2024-09-17T18:19:21.348 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../117-radosgw_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:19:21.377 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking radosgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:22.222 INFO:teuthology.orchestra.run.smithi105.stdout:Selecting previously unselected package rbd-fuse. 2024-09-17T18:19:22.239 INFO:teuthology.orchestra.run.smithi105.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4975-g1889a165-1jammy_amd64.deb ... 2024-09-17T18:19:22.280 INFO:teuthology.orchestra.run.smithi105.stdout:Unpacking rbd-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:22.568 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-17T18:19:22.818 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-17T18:19:22.945 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-17T18:19:23.204 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-17T18:19:23.564 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-17T18:19:24.001 INFO:teuthology.orchestra.run.smithi105.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-17T18:19:24.080 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:24.671 INFO:teuthology.orchestra.run.smithi105.stdout:Adding system user cephadm....done 2024-09-17T18:19:24.860 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-17T18:19:25.120 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-17T18:19:25.363 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-17T18:19:25.481 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-17T18:19:25.749 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-17T18:19:26.000 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-17T18:19:26.126 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-17T18:19:26.436 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-17T18:19:26.779 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-17T18:19:27.022 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-17T18:19:27.315 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-17T18:19:27.583 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-17T18:19:27.851 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-17T18:19:28.225 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-17T18:19:28.637 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-ceph-argparse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:28.879 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-17T18:19:28.988 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-17T18:19:29.114 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-17T18:19:29.240 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-17T18:19:29.366 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-17T18:19:29.617 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-17T18:19:30.041 INFO:teuthology.orchestra.run.smithi105.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-17T18:19:30.074 INFO:teuthology.orchestra.run.smithi105.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-17T18:19:30.145 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-17T18:19:30.255 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-17T18:19:30.598 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-17T18:19:30.723 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-17T18:19:30.993 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-17T18:19:31.259 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-17T18:19:31.568 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up zip (3.0-12build2) ... 2024-09-17T18:19:31.694 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-17T18:19:32.346 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-17T18:19:32.589 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-17T18:19:32.715 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-17T18:19:32.857 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-17T18:19:33.150 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-17T18:19:33.543 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-17T18:19:33.811 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-17T18:19:34.213 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-17T18:19:34.497 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-17T18:19:34.873 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-17T18:19:35.124 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-17T18:19:35.250 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:35.551 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-17T18:19:35.878 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-17T18:19:37.025 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-17T18:19:37.268 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T18:19:37.402 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-17T18:19:37.662 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-17T18:19:37.771 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-17T18:19:38.064 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-17T18:19:38.407 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-17T18:19:38.666 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-17T18:19:38.917 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T18:19:39.043 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-17T18:19:39.353 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-17T18:19:39.587 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-17T18:19:39.838 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-17T18:19:40.115 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-17T18:19:40.365 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-17T18:19:40.625 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-17T18:19:40.751 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-17T18:19:41.035 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-17T18:19:41.170 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-17T18:19:41.454 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-17T18:19:41.739 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-17T18:19:41.981 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-17T18:19:42.107 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-17T18:19:42.233 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-17T18:19:42.322 INFO:teuthology.orchestra.run.smithi105.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-17T18:19:42.568 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-17T18:19:42.694 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-17T18:19:43.079 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-17T18:19:43.330 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-17T18:19:43.456 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-17T18:19:43.732 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-17T18:19:43.858 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-17T18:19:44.176 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-17T18:19:44.477 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-17T18:19:44.761 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-17T18:19:44.879 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-17T18:19:45.146 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-17T18:19:45.527 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-17T18:19:45.792 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-17T18:19:46.169 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-17T18:19:46.445 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-17T18:19:46.773 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-17T18:19:46.899 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librados2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:47.025 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:47.168 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-17T18:19:47.512 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libcephfs2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:47.622 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libradosstriper1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:47.731 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-17T18:19:48.007 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librbd1 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:48.117 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-modules-core (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:48.218 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:48.435 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T18:19:48.435 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-17T18:19:48.936 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up libcephfs-dev (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:49.079 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rados (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:49.580 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up librgw2 (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:49.706 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-17T18:19:50.902 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rbd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:51.027 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up rbd-fuse (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:51.162 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-rgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:51.288 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up python3-cephfs (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:51.414 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-common (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:51.914 INFO:teuthology.orchestra.run.smithi105.stdout:Adding group ceph....done 2024-09-17T18:19:52.308 INFO:teuthology.orchestra.run.smithi105.stdout:Adding system user ceph....done 2024-09-17T18:19:52.477 INFO:teuthology.orchestra.run.smithi105.stdout:Setting system user ceph properties....done 2024-09-17T18:19:52.493 INFO:teuthology.orchestra.run.smithi105.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-17T18:19:52.630 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-17T18:19:52.931 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-17T18:19:53.415 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-test (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:53.532 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up radosgw (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:53.987 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T18:19:53.988 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-17T18:19:54.455 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-base (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:54.815 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-17T18:19:55.295 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mds (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:55.513 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T18:19:55.513 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-17T18:19:56.005 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:56.215 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T18:19:56.215 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-17T18:19:56.744 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-osd (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:57.104 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T18:19:57.104 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-17T18:19:57.551 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:57.652 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:57.804 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mon (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:58.021 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T18:19:58.021 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-17T18:19:58.463 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-cephadm (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:58.606 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:58.723 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-mgr-dashboard (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:58.866 INFO:teuthology.orchestra.run.smithi105.stdout:Setting up ceph-volume (19.3.0-4975-g1889a165-1jammy) ... 2024-09-17T18:19:59.219 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-17T18:19:59.311 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-17T18:19:59.885 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-17T18:20:00.660 INFO:teuthology.orchestra.run.smithi105.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-17T18:20:01.316 INFO:teuthology.orchestra.run.smithi105.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-17T18:20:01.322 DEBUG:teuthology.parallel:result is None 2024-09-17T18:20:01.322 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:20:01.477 DEBUG:teuthology.orchestra.run.smithi105:> dpkg-query -W -f '${Version}' ceph 2024-09-17T18:20:01.495 INFO:teuthology.orchestra.run.smithi105.stdout:19.3.0-4975-g1889a165-1jammy 2024-09-17T18:20:01.496 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4975-g1889a165-1jammy 2024-09-17T18:20:01.496 INFO:teuthology.task.install:The correct ceph version 19.3.0-4975-g1889a165-1jammy is installed. 2024-09-17T18:20:01.498 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T18:20:01.498 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:20:01.498 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T18:20:01.549 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T18:20:01.549 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:20:01.549 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T18:20:01.606 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T18:20:01.661 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T18:20:01.662 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:20:01.662 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T18:20:01.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T18:20:01.777 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T18:20:01.777 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:20:01.777 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T18:20:01.832 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T18:20:01.885 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T18:20:01.899 INFO:tasks.workunit:Pulling workunits from ref 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:20:01.900 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T18:20:01.900 INFO:tasks.workunit:timeout=3h 2024-09-17T18:20:01.900 INFO:tasks.workunit:cleanup=True 2024-09-17T18:20:01.900 DEBUG:teuthology.orchestra.run.smithi105:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-17T18:20:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T18:20:01.930 INFO:teuthology.orchestra.run.smithi105.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-17T18:20:01.930 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-17T18:20:01.977 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-17T18:20:01.978 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-17T18:20:02.026 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1889a165b5128787ec693450a3d1c5df5d6dc203 2024-09-17T18:20:02.076 INFO:tasks.workunit.client.0.smithi105.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-17T18:20:50.739 INFO:tasks.workunit.client.0.smithi105.stderr:Note: switching to '1889a165b5128787ec693450a3d1c5df5d6dc203'. 2024-09-17T18:20:50.739 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.739 INFO:tasks.workunit.client.0.smithi105.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: git switch -c 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:Or undo this operation with: 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: git switch - 2024-09-17T18:20:50.740 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.741 INFO:tasks.workunit.client.0.smithi105.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T18:20:50.741 INFO:tasks.workunit.client.0.smithi105.stderr: 2024-09-17T18:20:50.741 INFO:tasks.workunit.client.0.smithi105.stderr:HEAD is now at 1889a165b51 Merge branch 'wip-matanb-cleanup-snapmap' of https://github.com/Matan-B/ceph into wip-bharath10-testing-2024-09-17-1348 2024-09-17T18:20:50.745 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-17T18:20:50.795 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-09-17T18:20:50.795 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-17T18:20:50.841 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-09-17T18:20:50.842 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-09-17T18:20:50.842 DEBUG:teuthology.orchestra.run.smithi105:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-09-17T18:20:50.911 INFO:tasks.workunit.client.0.smithi105.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:20:50.917 INFO:tasks.workunit.client.0.smithi105.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:20:50.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:20:50.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:20:50.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:20:50.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:20:50.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-mapper 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-09-17T18:20:50.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-09-17T18:20:50.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none ' 2024-09-17T18:20:50.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-09-17T18:20:50.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-17T18:20:50.919 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-09-17T18:20:50.919 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T18:20:50.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-09-17T18:20:50.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:20:50.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:20:50.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:20:50.923 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:20:50.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:20:50.924 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:20:50.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:20:50.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:20:50.925 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:20:50.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:50.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:20:50.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:20:50.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:20:50.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:20:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:20:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-17T18:20:50.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:20:50.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:50.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:20:50.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21552 2024-09-17T18:20:50.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:20:50.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:20:50.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2024-09-17T18:20:50.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:20:50.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:20:50.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:20:50.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21552 2024-09-17T18:20:50.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:20:50.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T18:20:50.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T18:20:50.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:20:50.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-09-17T18:20:50.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-09-17T18:20:50.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-09-17T18:20:50.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-09-17T18:20:50.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-09-17T18:20:50.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-09-17T18:20:50.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-09-17T18:20:50.937 INFO:tasks.workunit.client.0.smithi105.stdout:Dir: td/osd-mapper 2024-09-17T18:20:50.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-09-17T18:20:50.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-09-17T18:20:50.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=test 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=test 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local extra_pars= 2024-09-17T18:20:50.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-17T18:20:50.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-17T18:20:50.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-17T18:20:52.807 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 c58cc0c9-41a1-49e0-80c4-86ea149386c7 2024-09-17T18:20:53.071 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:20:53.213 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:53.211+0000 7f5bbd8ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:53.216 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:53.211+0000 7f5bbd8ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:53.218 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:53.215+0000 7f5bbd8ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:53.218 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:53.215+0000 7f5bbd8ae7c0 -1 bdev(0x5613272afc00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:20:53.218 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:53.215+0000 7f5bbd8ae7c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-09-17T18:20:59.079 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:20:59.284 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:20:59.316 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:59.311+0000 7ff27f13e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:59.318 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:59.315+0000 7ff27f13e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:59.320 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:20:59.315+0000 7ff27f13e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:20:59.480 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:21:00.427 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:00.423+0000 7ff27f13e7c0 -1 Falling back to public interface 2024-09-17T18:21:00.673 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:21:01.860 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:21:03.187 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:21:03.243 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:03.239+0000 7ff27f13e7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:21:04.680 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:21:06.003 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:21:07.337 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:21:08.672 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:21:09.996 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:21:11.317 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:21:12.641 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:21:14.002 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:21:15.332 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:21:16.872 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:21:18.249 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:21:19.624 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:21:21.025 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:21:22.714 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:21:24.408 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:21:26.111 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:21:27.799 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:21:28.145 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1605559083,v1:127.0.0.1:6801/1605559083] [v2:127.0.0.1:6802/1605559083,v1:127.0.0.1:6803/1605559083] exists,up c58cc0c9-41a1-49e0-80c4-86ea149386c7 2024-09-17T18:21:28.150 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 d8f0f09f-4b44-482c-819a-163a9b5fd04f 2024-09-17T18:21:28.565 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:21:28.621 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:28.619+0000 7f5cba7f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:28.641 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:28.639+0000 7f5cba7f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:28.643 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:28.639+0000 7f5cba7f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:28.644 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:28.639+0000 7f5cba7f07c0 -1 bdev(0x5649d933fc00 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:21:28.644 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:28.639+0000 7f5cba7f07c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-09-17T18:21:34.638 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:21:35.030 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:21:35.068 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:35.063+0000 7fcb7057a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:35.087 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:35.083+0000 7fcb7057a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:35.088 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:35.083+0000 7fcb7057a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:21:35.392 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:21:36.387 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:36.383+0000 7fcb7057a7c0 -1 Falling back to public interface 2024-09-17T18:21:36.725 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:21:38.046 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:21:39.318 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:21:39.315+0000 7fcb7057a7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:21:39.360 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:21:40.782 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:21:42.119 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:21:43.446 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:21:44.766 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:21:46.192 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:21:47.525 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:21:48.877 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:21:50.249 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:21:51.581 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:21:53.216 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:21:55.247 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:21:56.824 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:21:58.702 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:22:00.304 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:22:01.837 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:22:03.247 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:22:03.569 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2714957656,v1:127.0.0.1:6811/2714957656] [v2:127.0.0.1:6812/2714957656,v1:127.0.0.1:6813/2714957656] exists,up d8f0f09f-4b44-482c-819a-163a9b5fd04f 2024-09-17T18:22:03.574 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 88a9cd77-8565-4a91-ab48-1b6fe38124f5 2024-09-17T18:22:03.952 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:22:04.008 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:04.005+0000 7f67667ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:04.028 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:04.025+0000 7f67667ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:04.029 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:04.025+0000 7f67667ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:04.030 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:04.029+0000 7f67667ae7c0 -1 bdev(0x55977d0fdc00 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:22:04.030 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:04.029+0000 7f67667ae7c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-09-17T18:22:09.615 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:22:10.027 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:22:10.065 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:10.061+0000 7f2adb0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:10.074 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:10.073+0000 7f2adb0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:10.075 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:10.073+0000 7f2adb0b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:22:10.348 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:22:11.405 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:11.401+0000 7f2adb0b17c0 -1 Falling back to public interface 2024-09-17T18:22:11.676 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:22:12.999 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:22:14.312 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:22:14.529 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:14.525+0000 7f2adb0b17c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:22:15.638 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:22:16.965 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:22:18.395 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:22:19.725 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:22:21.096 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:22:22.586 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:22:23.919 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:22:25.440 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:22:26.771 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:22:28.185 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:22:29.658 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:22:31.102 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:22:32.456 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:22:34.213 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:22:35.975 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:22:37.417 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:22:37.795 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3538145075,v1:127.0.0.1:6819/3538145075] [v2:127.0.0.1:6820/3538145075,v1:127.0.0.1:6821/3538145075] exists,up 88a9cd77-8565-4a91-ab48-1b6fe38124f5 2024-09-17T18:22:38.234 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'test' created 2024-09-17T18:22:40.285 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836501 2024-09-17T18:22:41.970 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672973 2024-09-17T18:22:42.312 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509447 2024-09-17T18:22:45.109 INFO:tasks.workunit.client.0.smithi105.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-09-17T18:22:45.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-09-17T18:22:45.279 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0: { 2024-09-17T18:22:45.279 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-17T18:22:45.279 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:45.294 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1: { 2024-09-17T18:22:45.294 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-17T18:22:45.294 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:45.308 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2: { 2024-09-17T18:22:45.308 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-09-17T18:22:45.308 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:45.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-09-17T18:22:45.441 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0: { 2024-09-17T18:22:45.441 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-17T18:22:45.441 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:45.453 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1: { 2024-09-17T18:22:45.453 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-17T18:22:45.453 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:45.466 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2: { 2024-09-17T18:22:45.466 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-09-17T18:22:45.466 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-09-17T18:22:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-09-17T18:22:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-09-17T18:22:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-09-17T18:22:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-09-17T18:22:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-17T18:22:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-17T18:22:45.689 INFO:tasks.workunit.client.0.smithi105.stdout:created pool test snap snap01 2024-09-17T18:22:45.997 INFO:tasks.workunit.client.0.smithi105.stdout:created pool test snap snap02 2024-09-17T18:22:46.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-09-17T18:22:46.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-17T18:22:46.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-17T18:22:46.378 INFO:tasks.workunit.client.0.smithi105.stdout:created pool test snap snap13 2024-09-17T18:22:46.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-09-17T18:22:46.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-09-17T18:22:46.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-09-17T18:22:46.864 INFO:tasks.workunit.client.0.smithi105.stdout:created pool test snap snap24 2024-09-17T18:22:47.060 INFO:tasks.workunit.client.0.smithi105.stdout:created pool test snap snap25 2024-09-17T18:22:47.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 8175 2024-09-17T18:22:47.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-09-17T18:22:47.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-17T18:22:47.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-09-17T18:22:47.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-09-17T18:22:47.709 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-17T18:22:47.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-17T18:22:48.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-09-17T18:22:48.083 INFO:tasks.workunit.client.0.smithi105.stdout:pgid is 1.3 (primary: osd.1) 2024-09-17T18:22:48.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-09-17T18:22:48.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-09-17T18:22:48.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:306: set_query_debug: local pgid=1.3 2024-09-17T18:22:48.084 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: ceph pg dump pgs_brief 2024-09-17T18:22:48.084 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-09-17T18:22:48.402 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs_brief 2024-09-17T18:22:48.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: set_query_debug: local prim_osd=1 2024-09-17T18:22:48.420 INFO:tasks.workunit.client.0.smithi105.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-09-17T18:22:48.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:310: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-09-17T18:22:48.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: get_asok_path osd.1 2024-09-17T18:22:48.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:22:48.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:22:48.421 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:22:48.421 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:22:48.421 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:22:48.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21552/ceph-osd.1.asok 2024-09-17T18:22:48.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: CEPH_ARGS= 2024-09-17T18:22:48.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:311: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.21552/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-09-17T18:22:48.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:22:48.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout:{"success":true}{ 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "objxxx", 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: "clones": [ 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 2, 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [ 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 1, 2024-09-17T18:22:48.591 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap01" 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 2, 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap02" 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: ], 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 6, 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: "overlaps": [] 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 3, 2024-09-17T18:22:48.592 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [ 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 3, 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap13" 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: ], 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 6, 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: "overlaps": [] 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 5, 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [ 2024-09-17T18:22:48.593 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 4, 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap24" 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 5, 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap25" 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: ], 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 6, 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "overlaps": [] 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:22:48.594 INFO:tasks.workunit.client.0.smithi105.stdout: "id": "head", 2024-09-17T18:22:48.595 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [], 2024-09-17T18:22:48.595 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 5 2024-09-17T18:22:48.595 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:22:48.595 INFO:tasks.workunit.client.0.smithi105.stdout: ] 2024-09-17T18:22:48.595 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:48.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-17T18:22:48.711 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:22:48.711 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": true, 2024-09-17T18:22:48.711 INFO:tasks.workunit.client.0.smithi105.stdout: "must": true, 2024-09-17T18:22:48.711 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "0.000000" 2024-09-17T18:22:48.711 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:22:48.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-09-17T18:22:51.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-17T18:22:39.190878+0000 27'7 27:38 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:22:39.358046+0000 0'0 27:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:22:39.358009+0000 0'0 27:33 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:22:39.191013+0000 0'0 27:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:22:52.030 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:22:52.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-09-17T18:22:52.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-17T18:22:39.190878+0000 27'7 27:38 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:22:39.358046+0000 0'0 27:31 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:22:39.358009+0000 0'0 27:33 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:22:39.191013+0000 0'0 27:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:22:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:22:52.348 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:22:52.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-09-17T18:22:52.760 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T18:22:52.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-09-17T18:22:53.220 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T18:22:53.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-09-17T18:22:58.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-09-17T18:22:58.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-17T18:22:58.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:22:58.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:22:58.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:22:58.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:22:58.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:22:58.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:22:58.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:22:58.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-17T18:22:58.588 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.583+0000 7f28d0a57ac0 1 bdev(0x559629d5d800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:22:58.588 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.583+0000 7f28d0a57ac0 1 bdev(0x559629d5d800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:58.649 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.643+0000 7f28d0a57ac0 1 bdev(0x559629d5d800 td/osd-mapper/0/block) close 2024-09-17T18:22:58.867 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bdev(0x559629d5d800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:22:58.867 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bdev(0x559629d5d800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:58.868 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-17T18:22:58.868 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bdev(0x55962a0aa000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:22:58.869 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bdev(0x55962a0aa000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:58.869 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-17T18:22:58.869 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bluefs mount 2024-09-17T18:22:58.869 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.863+0000 7f28d0a57ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:22:58.895 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.887+0000 7f28d0a57ac0 1 bluefs mount shared_bdev_used = 0 2024-09-17T18:22:58.919 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.887+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 freelist init 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 freelist _read_cfg 2024-09-17T18:22:58.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-17T18:22:58.962 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 8.39236e-07 2024-09-17T18:22:58.962 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bluefs umount 2024-09-17T18:22:58.962 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:58.955+0000 7f28d0a57ac0 1 bdev(0x55962a0aa000 td/osd-mapper/0/block) close 2024-09-17T18:22:59.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.147+0000 7f28d0a57ac0 1 bdev(0x55962a0aa000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:22:59.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.147+0000 7f28d0a57ac0 1 bdev(0x55962a0aa000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:59.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.147+0000 7f28d0a57ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-17T18:22:59.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.147+0000 7f28d0a57ac0 1 bluefs mount 2024-09-17T18:22:59.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.147+0000 7f28d0a57ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:22:59.158 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.151+0000 7f28d0a57ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-17T18:22:59.158 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.151+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:22:59.207 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.203+0000 7f28d0a57ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:22:59.208 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-17T18:22:59.208 INFO:tasks.workunit.client.0.smithi105.stdout:00000000 0b 05 58 02 00 00 13 0e b6 d4 59 c2 4b d5 b8 c6 |..X.......Y.K...| 2024-09-17T18:22:59.208 INFO:tasks.workunit.client.0.smithi105.stdout:00000010 3c 74 78 89 38 f9 00 00 00 00 1d 00 00 00 00 00 |= 2 )) 2024-09-17T18:22:59.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-17T18:22:59.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-09-17T18:22:59.683 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.679+0000 7f0f21330ac0 1 bdev(0x55dafa899800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:22:59.683 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.679+0000 7f0f21330ac0 1 bdev(0x55dafa899800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:59.700 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.695+0000 7f0f21330ac0 1 bdev(0x55dafa899800 td/osd-mapper/2/block) close 2024-09-17T18:22:59.971 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bdev(0x55dafa899800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:22:59.971 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bdev(0x55dafa899800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:59.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-17T18:22:59.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bdev(0x55dafabe6000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:22:59.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bdev(0x55dafabe6000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:22:59.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-17T18:22:59.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bluefs mount 2024-09-17T18:22:59.973 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.967+0000 7f0f21330ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:22:59.987 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.983+0000 7f0f21330ac0 1 bluefs mount shared_bdev_used = 0 2024-09-17T18:22:59.987 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:22:59.983+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:00.048 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:00.048 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 freelist init 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 freelist _read_cfg 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-17T18:23:00.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 4.95912e-07 2024-09-17T18:23:00.050 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bluefs umount 2024-09-17T18:23:00.050 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.043+0000 7f0f21330ac0 1 bdev(0x55dafabe6000 td/osd-mapper/2/block) close 2024-09-17T18:23:00.267 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.263+0000 7f0f21330ac0 1 bdev(0x55dafabe6000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:23:00.267 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.263+0000 7f0f21330ac0 1 bdev(0x55dafabe6000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:00.267 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.263+0000 7f0f21330ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-17T18:23:00.267 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.263+0000 7f0f21330ac0 1 bluefs mount 2024-09-17T18:23:00.267 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.263+0000 7f0f21330ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:00.273 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.267+0000 7f0f21330ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-17T18:23:00.273 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.267+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:00.322 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.315+0000 7f0f21330ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:00.322 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.315+0000 7f0f21330ac0 1 bluefs umount 2024-09-17T18:23:00.322 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.315+0000 7f0f21330ac0 1 bdev(0x55dafabe6000 td/osd-mapper/2/block) close 2024-09-17T18:23:00.547 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.543+0000 7f0f21330ac0 1 freelist shutdown 2024-09-17T18:23:00.547 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:00.543+0000 7f0f21330ac0 1 bdev(0x55dafa899800 td/osd-mapper/2/block) close 2024-09-17T18:23:00.761 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-17T18:23:00.762 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-17T18:23:00.762 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:00.762 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-17T18:23:00.762 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-17T18:23:00.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:00.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-09-17T18:23:00.763 INFO:tasks.workunit.client.0.smithi105.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-09-17T18:23:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-17T18:23:01.447 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-17T18:23:01.447 INFO:tasks.workunit.client.0.smithi105.stdout:00000000 0b 05 58 02 00 00 13 0e b6 d4 59 c2 4b d5 b8 c6 |..X.......Y.K...| 2024-09-17T18:23:01.447 INFO:tasks.workunit.client.0.smithi105.stdout:00000010 3c 74 78 89 38 f9 02 00 00 00 1d 00 00 00 00 00 |= 3 )) 2024-09-17T18:23:02.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-09-17T18:23:02.991 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:02.987+0000 7fef7a2b7ac0 1 bdev(0x560ae8875800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:23:02.991 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:02.987+0000 7fef7a2b7ac0 1 bdev(0x560ae8875800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:02.992 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:02.987+0000 7fef7a2b7ac0 1 bdev(0x560ae8875800 td/osd-mapper/0/block) close 2024-09-17T18:23:03.271 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bdev(0x560ae8875800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:23:03.271 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bdev(0x560ae8875800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:03.272 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-17T18:23:03.272 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bdev(0x560ae8bc2000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:23:03.272 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bdev(0x560ae8bc2000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:03.272 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-17T18:23:03.273 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bluefs mount 2024-09-17T18:23:03.273 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.267+0000 7fef7a2b7ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:03.280 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.275+0000 7fef7a2b7ac0 1 bluefs mount shared_bdev_used = 0 2024-09-17T18:23:03.280 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.275+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-09-17T18:23:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-09-17T18:23:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-17T18:23:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-09-17T18:23:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 freelist init 2024-09-17T18:23:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 freelist _read_cfg 2024-09-17T18:23:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-17T18:23:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 23 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 8.39236e-07 2024-09-17T18:23:03.335 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bluefs umount 2024-09-17T18:23:03.335 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.327+0000 7fef7a2b7ac0 1 bdev(0x560ae8bc2000 td/osd-mapper/0/block) close 2024-09-17T18:23:03.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.551+0000 7fef7a2b7ac0 1 bdev(0x560ae8bc2000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-09-17T18:23:03.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.551+0000 7fef7a2b7ac0 1 bdev(0x560ae8bc2000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:03.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.551+0000 7fef7a2b7ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-09-17T18:23:03.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.551+0000 7fef7a2b7ac0 1 bluefs mount 2024-09-17T18:23:03.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.551+0000 7fef7a2b7ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:03.562 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.555+0000 7fef7a2b7ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-17T18:23:03.562 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.555+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:03.608 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:03.603+0000 7fef7a2b7ac0 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:03.609 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-17T18:23:03.609 INFO:tasks.workunit.client.0.smithi105.stdout:00000000 0b 05 58 02 00 00 13 0e b6 d4 59 c2 4b d5 b8 c6 |..X.......Y.K...| 2024-09-17T18:23:03.609 INFO:tasks.workunit.client.0.smithi105.stdout:00000010 3c 74 78 89 38 f9 00 00 00 00 1d 00 00 00 00 00 |= 1 )) 2024-09-17T18:23:05.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-17T18:23:05.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-17T18:23:05.155 INFO:tasks.workunit.client.0.smithi105.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:05.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-17T18:23:05.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.LhScHCAueM_the_val 2024-09-17T18:23:05.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:05.159 INFO:tasks.workunit.client.0.smithi105.stdout:Value dumped in: /tmp/tmp.LhScHCAueM_the_val 2024-09-17T18:23:05.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.LhScHCAueM_the_val' 2024-09-17T18:23:05.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.LhScHCAueM_the_val 2024-09-17T18:23:05.821 INFO:tasks.workunit.client.0.smithi105.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-17T18:23:06.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:06.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.LhScHCAueM_the_val 2024-09-17T18:23:06.289 INFO:tasks.workunit.client.0.smithi105.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-17T18:23:06.290 INFO:tasks.workunit.client.0.smithi105.stdout:0000073 2024-09-17T18:23:06.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-17T18:23:06.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:08.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.LhScHCAueM_the_val 2024-09-17T18:23:09.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:09.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:09.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-17T18:23:09.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-09-17T18:23:09.493 INFO:tasks.workunit.client.0.smithi105.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-09-17T18:23:09.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-17T18:23:09.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-17T18:23:09.521 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.515+0000 7fc6c17a2ac0 1 bdev(0x555a10f39800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-17T18:23:09.521 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.515+0000 7fc6c17a2ac0 1 bdev(0x555a10f39800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:09.529 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.523+0000 7fc6c17a2ac0 1 bdev(0x555a10f39800 td/osd-mapper/1/block) close 2024-09-17T18:23:09.803 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bdev(0x555a10f39800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-17T18:23:09.803 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bdev(0x555a10f39800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:09.804 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-17T18:23:09.804 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bdev(0x555a11286000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-17T18:23:09.804 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bdev(0x555a11286000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:09.804 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-17T18:23:09.804 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bluefs mount 2024-09-17T18:23:09.805 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.799+0000 7fc6c17a2ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:09.812 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.807+0000 7fc6c17a2ac0 1 bluefs mount shared_bdev_used = 0 2024-09-17T18:23:09.812 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.807+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:09.869 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 freelist init 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 freelist _read_cfg 2024-09-17T18:23:09.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-17T18:23:09.871 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 18 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 6.485e-07 2024-09-17T18:23:09.871 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bluefs umount 2024-09-17T18:23:09.871 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:09.863+0000 7fc6c17a2ac0 1 bdev(0x555a11286000 td/osd-mapper/1/block) close 2024-09-17T18:23:10.083 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.079+0000 7fc6c17a2ac0 1 bdev(0x555a11286000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-09-17T18:23:10.083 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.079+0000 7fc6c17a2ac0 1 bdev(0x555a11286000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:10.083 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.079+0000 7fc6c17a2ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-09-17T18:23:10.084 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.079+0000 7fc6c17a2ac0 1 bluefs mount 2024-09-17T18:23:10.094 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.091+0000 7fc6c17a2ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:10.103 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.099+0000 7fc6c17a2ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-17T18:23:10.104 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.099+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:10.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:10.143+0000 7fc6c17a2ac0 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:10.152 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-17T18:23:10.152 INFO:tasks.workunit.client.0.smithi105.stdout:00000000 0b 05 58 02 00 00 13 0e b6 d4 59 c2 4b d5 b8 c6 |..X.......Y.K...| 2024-09-17T18:23:10.152 INFO:tasks.workunit.client.0.smithi105.stdout:00000010 3c 74 78 89 38 f9 01 00 00 00 1d 00 00 00 00 00 |= 1 )) 2024-09-17T18:23:11.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-17T18:23:11.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-17T18:23:11.695 INFO:tasks.workunit.client.0.smithi105.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:11.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-17T18:23:11.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.pRn5cVuaiz_the_val 2024-09-17T18:23:11.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:11.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.pRn5cVuaiz_the_val' 2024-09-17T18:23:11.698 INFO:tasks.workunit.client.0.smithi105.stdout:Value dumped in: /tmp/tmp.pRn5cVuaiz_the_val 2024-09-17T18:23:11.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.pRn5cVuaiz_the_val 2024-09-17T18:23:12.374 INFO:tasks.workunit.client.0.smithi105.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-17T18:23:12.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:12.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.pRn5cVuaiz_the_val 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-17T18:23:12.803 INFO:tasks.workunit.client.0.smithi105.stdout:0000073 2024-09-17T18:23:12.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-17T18:23:12.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:14.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.pRn5cVuaiz_the_val 2024-09-17T18:23:16.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:16.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:16.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-09-17T18:23:16.010 INFO:tasks.workunit.client.0.smithi105.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-09-17T18:23:16.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-09-17T18:23:16.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-09-17T18:23:16.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-09-17T18:23:16.037 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.031+0000 7fb0dc8f0ac0 1 bdev(0x557d5f69b800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:23:16.037 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.031+0000 7fb0dc8f0ac0 1 bdev(0x557d5f69b800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:16.048 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.043+0000 7fb0dc8f0ac0 1 bdev(0x557d5f69b800 td/osd-mapper/2/block) close 2024-09-17T18:23:16.323 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bdev(0x557d5f69b800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:23:16.323 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bdev(0x557d5f69b800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:16.324 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-09-17T18:23:16.324 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bdev(0x557d5f9e8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:23:16.324 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bdev(0x557d5f9e8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:16.325 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-17T18:23:16.325 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bluefs mount 2024-09-17T18:23:16.325 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.319+0000 7fb0dc8f0ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:16.343 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.339+0000 7fb0dc8f0ac0 1 bluefs mount shared_bdev_used = 0 2024-09-17T18:23:16.343 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.339+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 freelist init 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 freelist _read_cfg 2024-09-17T18:23:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.395+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-09-17T18:23:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.399+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 4.95912e-07 2024-09-17T18:23:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.399+0000 7fb0dc8f0ac0 1 bluefs umount 2024-09-17T18:23:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.399+0000 7fb0dc8f0ac0 1 bdev(0x557d5f9e8000 td/osd-mapper/2/block) close 2024-09-17T18:23:16.607 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.603+0000 7fb0dc8f0ac0 1 bdev(0x557d5f9e8000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-09-17T18:23:16.607 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.603+0000 7fb0dc8f0ac0 1 bdev(0x557d5f9e8000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-09-17T18:23:16.607 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.603+0000 7fb0dc8f0ac0 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-09-17T18:23:16.607 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.603+0000 7fb0dc8f0ac0 1 bluefs mount 2024-09-17T18:23:16.617 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.611+0000 7fb0dc8f0ac0 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-09-17T18:23:16.625 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.619+0000 7fb0dc8f0ac0 1 bluefs mount shared_bdev_used = 27459584 2024-09-17T18:23:16.626 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.619+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-09-17T18:23:16.671 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:16.667+0000 7fb0dc8f0ac0 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-09-17T18:23:16.673 INFO:tasks.workunit.client.0.smithi105.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-09-17T18:23:16.673 INFO:tasks.workunit.client.0.smithi105.stdout:00000000 0b 05 58 02 00 00 13 0e b6 d4 59 c2 4b d5 b8 c6 |..X.......Y.K...| 2024-09-17T18:23:16.673 INFO:tasks.workunit.client.0.smithi105.stdout:00000010 3c 74 78 89 38 f9 02 00 00 00 1d 00 00 00 00 00 |= 1 )) 2024-09-17T18:23:18.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-09-17T18:23:18.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-09-17T18:23:18.207 INFO:tasks.workunit.client.0.smithi105.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:18.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-09-17T18:23:18.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.DQ4MktGAts_the_val 2024-09-17T18:23:18.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:18.210 INFO:tasks.workunit.client.0.smithi105.stdout:Value dumped in: /tmp/tmp.DQ4MktGAts_the_val 2024-09-17T18:23:18.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.DQ4MktGAts_the_val' 2024-09-17T18:23:18.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.DQ4MktGAts_the_val 2024-09-17T18:23:18.877 INFO:tasks.workunit.client.0.smithi105.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-09-17T18:23:19.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:19.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.DQ4MktGAts_the_val 2024-09-17T18:23:19.326 INFO:tasks.workunit.client.0.smithi105.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 � u � 205 \0 \0 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-09-17T18:23:19.327 INFO:tasks.workunit.client.0.smithi105.stdout:0000073 2024-09-17T18:23:19.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-17T18:23:19.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:21.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.DQ4MktGAts_the_val 2024-09-17T18:23:22.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:22.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 ' 2024-09-17T18:23:22.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 2024-09-17T18:23:22.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:22.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:22.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/' 2024-09-17T18:23:22.782 INFO:tasks.workunit.client.0.smithi105.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3/ 2024-09-17T18:23:22.783 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-09-17T18:23:22.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:23:22.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:22.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:22.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:22.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:23:22.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21552/$cluster-$name.asok' 2024-09-17T18:23:22.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21552/$cluster-$name.asok' 2024-09-17T18:23:22.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:23:22.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-17T18:23:22.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-17T18:23:22.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:23:22.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:23:22.792 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:23:22.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.21552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:23:22.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2024-09-17T18:23:22.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:23:22.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:23:22.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:23:22.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:23:22.844 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:22.838+0000 7f1bcf0ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:22.847 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:22.842+0000 7f1bcf0ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:22.848 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:22.842+0000 7f1bcf0ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:23.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:23:23.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:23:23.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:23:23.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:23:23.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:23:23.149 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:23:23.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:23.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:23:23.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:23.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:23.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:24.154 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:24.146+0000 7f1bcf0ba7c0 -1 Falling back to public interface 2024-09-17T18:23:24.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:24.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:24.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:23:24.528 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:23:24.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:24.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:24.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:25.894 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:23:25.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:25.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:25.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:23:25.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:25.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:26.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:26.806 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:26.798+0000 7f1bcf0ba7c0 -1 osd.0 29 log_to_monitors true 2024-09-17T18:23:27.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:27.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:27.273 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:23:27.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:23:27.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:27.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:23:27.634 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 32 up_thru 32 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6800/77762192,v1:127.0.0.1:6801/77762192] [v2:127.0.0.1:6802/77762192,v1:127.0.0.1:6803/77762192] exists,up c58cc0c9-41a1-49e0-80c4-86ea149386c7 2024-09-17T18:23:27.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:23:27.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:23:27.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21552/$cluster-$name.asok' 2024-09-17T18:23:27.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21552/$cluster-$name.asok' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:23:27.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2024-09-17T18:23:27.641 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:23:27.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:23:27.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.21552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:23:27.642 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2024-09-17T18:23:27.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:23:27.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:23:27.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:23:27.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:23:27.689 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:27.682+0000 7fef800ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:27.745 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:27.738+0000 7fef800ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:27.746 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:27.738+0000 7fef800ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:27.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:23:27.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:23:27.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:23:27.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:23:27.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:23:27.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:27.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:23:27.985 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:23:27.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:27.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:28.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:28.805 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:28.798+0000 7fef800ea7c0 -1 Falling back to public interface 2024-09-17T18:23:29.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:29.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:29.334 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:23:29.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:23:29.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:29.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:29.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:30.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:30.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:30.693 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:23:30.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:23:30.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:30.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:31.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:31.451 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:31.446+0000 7fef800ea7c0 -1 osd.1 29 log_to_monitors true 2024-09-17T18:23:32.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:32.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:32.051 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:23:32.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:23:32.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:32.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:23:32.523 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/3117150052,v1:127.0.0.1:6811/3117150052] [v2:127.0.0.1:6812/3117150052,v1:127.0.0.1:6813/3117150052] exists,up d8f0f09f-4b44-482c-819a-163a9b5fd04f 2024-09-17T18:23:32.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3' 2024-09-17T18:23:32.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:32.528 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21552/$cluster-$name.asok' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21552/$cluster-$name.asok' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:23:32.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:23:32.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:23:32.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2024-09-17T18:23:32.530 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=130eb6d4-59c2-4bd5-b8c6-3c74788938f9 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.21552/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:23:32.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:23:32.579 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:32.574+0000 7faa8268b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:32.657 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:32.650+0000 7faa8268b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:32.658 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:32.650+0000 7faa8268b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:32.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:23:32.947 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:23:32.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:32.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:23:33.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:33.745 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:33.738+0000 7faa8268b7c0 -1 Falling back to public interface 2024-09-17T18:23:34.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:34.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:34.300 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:23:34.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:23:34.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:34.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:23:34.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:35.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:35.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:35.656 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:23:35.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:23:35.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:35.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:23:36.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:23:36.423 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:36.418+0000 7faa8268b7c0 -1 osd.2 29 log_to_monitors true 2024-09-17T18:23:37.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:23:37.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:23:37.023 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:23:37.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:23:37.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:23:37.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:23:37.479 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 39 up_thru 16 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6818/417005017,v1:127.0.0.1:6819/417005017] [v2:127.0.0.1:6820/417005017,v1:127.0.0.1:6821/417005017] exists,up 88a9cd77-8565-4a91-ab48-1b6fe38124f5 2024-09-17T18:23:37.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:23:37.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:23:37.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:23:37.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-09-17T18:23:38.483 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-09-17T18:23:38.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-09-17T18:23:38.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:38.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-09-17T18:23:38.626 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:23:38.626 INFO:tasks.workunit.client.0.smithi105.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:23:38.626 INFO:tasks.workunit.client.0.smithi105.stdout: "release": "squid", 2024-09-17T18:23:38.626 INFO:tasks.workunit.client.0.smithi105.stdout: "release_type": "dev" 2024-09-17T18:23:38.626 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:23:38.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:38.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-09-17T18:23:38.767 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:23:38.767 INFO:tasks.workunit.client.0.smithi105.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:23:38.768 INFO:tasks.workunit.client.0.smithi105.stdout: "release": "squid", 2024-09-17T18:23:38.768 INFO:tasks.workunit.client.0.smithi105.stdout: "release_type": "dev" 2024-09-17T18:23:38.768 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:23:38.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-09-17T18:23:38.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-09-17T18:23:38.916 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:23:38.916 INFO:tasks.workunit.client.0.smithi105.stdout: "version": "19.3.0-4975-g1889a165", 2024-09-17T18:23:38.916 INFO:tasks.workunit.client.0.smithi105.stdout: "release": "squid", 2024-09-17T18:23:38.916 INFO:tasks.workunit.client.0.smithi105.stdout: "release_type": "dev" 2024-09-17T18:23:38.917 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:23:38.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-09-17T18:23:38.959 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:23:38.959 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "objxxx", 2024-09-17T18:23:38.959 INFO:tasks.workunit.client.0.smithi105.stdout: "clones": [ 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 2, 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [ 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 1, 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap01" 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 2, 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap02" 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:23:38.960 INFO:tasks.workunit.client.0.smithi105.stdout: ], 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 6, 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "overlaps": [] 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 3, 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [ 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 3, 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap13" 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: ], 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 6, 2024-09-17T18:23:38.961 INFO:tasks.workunit.client.0.smithi105.stdout: "overlaps": [] 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 5, 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [ 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 4, 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap24" 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "id": 5, 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "name": "snap25" 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: ], 2024-09-17T18:23:38.962 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 6, 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: "overlaps": [] 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: }, 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: { 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: "id": "head", 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: "snapshots": [], 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: "size": 5 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: } 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout: ] 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:23:38.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-09-17T18:23:39.403 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is unset 2024-09-17T18:23:39.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-09-17T18:23:39.955 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is unset 2024-09-17T18:23:39.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-09-17T18:23:39.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-09-17T18:23:40.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-09-17T18:23:40.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-17T18:23:40.637 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:23:40.637 INFO:tasks.workunit.client.0.smithi105.stdout:1.3 4 0 4 0 0 23 0 0 7 0 7 active+undersized+degraded 2024-09-17T18:23:32.618885+0000 27'7 37:57 [1,0] 1 [1,0] 1 27'7 2024-09-17T18:22:49.380307+0000 27'7 2024-09-17T18:22:49.380307+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:22:49.380307+0000 4 0 2024-09-17T18:23:40.637 INFO:tasks.workunit.client.0.smithi105.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 activating 2024-09-17T18:23:37.221416+0000 0'0 40:35 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:23:40.638 INFO:tasks.workunit.client.0.smithi105.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2024-09-17T18:23:32.024546+0000 0'0 35:13 [0,1] 0 [0,1] 0 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:23:40.638 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-09-17T18:23:32.618439+0000 0'0 37:23 [1,0] 1 [1,0] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:40.638 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:23:40.638 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:23:40.638 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:23:40.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-09-17T18:23:42.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-17T18:23:42.794 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:23:42.794 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": true, 2024-09-17T18:23:42.794 INFO:tasks.workunit.client.0.smithi105.stdout: "must": true, 2024-09-17T18:23:42.794 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "0.000000" 2024-09-17T18:23:42.795 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:23:42.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-09-17T18:23:47.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-17T18:23:43.403218+0000 27'7 43:82 [1,2,0] 1 [1,2,0] 1 27'7 2024-09-17T18:23:43.403092+0000 27'7 2024-09-17T18:23:43.403092+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:23:43.403092+0000 4 0 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:23:37.514503+0000 0'0 43:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:23:37.985291+0000 0'0 43:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:23:37.984975+0000 0'0 43:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:23:48.143 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:23:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-17T18:23:48.161 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:23:43.398+0000 7fef62871640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-17T18:23:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-09-17T18:23:48.164 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-17T18:23:48.164 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-09-17T18:23:48.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-09-17T18:23:48.167 INFO:tasks.workunit.client.0.smithi105.stdout:prev count: 1 2024-09-17T18:23:48.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-09-17T18:23:48.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-09-17T18:23:48.293 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:23:48.293 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": true, 2024-09-17T18:23:48.293 INFO:tasks.workunit.client.0.smithi105.stdout: "must": true, 2024-09-17T18:23:48.293 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "0.000000" 2024-09-17T18:23:48.293 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:23:48.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-09-17T18:23:53.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-09-17T18:23:53.620 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:23:53.620 INFO:tasks.workunit.client.0.smithi105.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-09-17T18:23:48.422625+0000 27'7 43:90 [1,2,0] 1 [1,2,0] 1 27'7 2024-09-17T18:23:48.422498+0000 27'7 2024-09-17T18:23:48.422498+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:23:48.422498+0000 4 0 2024-09-17T18:23:53.620 INFO:tasks.workunit.client.0.smithi105.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:23:37.514503+0000 0'0 43:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:53.621 INFO:tasks.workunit.client.0.smithi105.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:23:37.985291+0000 0'0 43:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:53.621 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:23:37.984975+0000 0'0 43:39 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-17T18:22:38.178031+0000 0'0 2024-09-17T18:22:38.178031+0000 0 0 periodic scrub scheduled @ 2024-09-18T18:22:38.178031+0000 0 0 2024-09-17T18:23:53.621 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:23:53.621 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:23:53.621 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:23:53.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:53.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-17T18:23:53.640 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:23:43.398+0000 7fef62871640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-09-17T18:23:53.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-09-17T18:23:53.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-09-17T18:23:53.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-09-17T18:23:53.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-09-17T18:23:53.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-09-17T18:23:53.644 INFO:tasks.workunit.client.0.smithi105.stdout:current count: 1 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:23:53.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:23:53.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:23:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-09-17T18:23:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-09-17T18:23:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-17T18:23:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-09-17T18:23:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-17T18:23:55.047 INFO:tasks.workunit.client.0.smithi105.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-09-17T18:23:55.047 INFO:tasks.workunit.client.0.smithi105.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-09-17T18:23:55.047 INFO:tasks.workunit.client.0.smithi105.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-09-17T18:23:55.047 INFO:tasks.workunit.client.0.smithi105.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-09-17T18:23:55.047 INFO:tasks.workunit.client.0.smithi105.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-09-17T18:23:55.047 INFO:tasks.workunit.client.0.smithi105.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-09-17T18:23:55.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-09-17T18:23:55.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-09-17T18:23:55.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-09-17T18:23:55.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-09-17T18:23:56.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-09-17T18:23:56.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-09-17T18:23:56.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-09-17T18:23:56.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-09-17T18:23:56.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-17T18:23:56.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:23:56.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-17T18:23:56.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:23:56.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:23:56.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:23:56.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:23:56.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:23:56.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:23:56.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:23:56.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:23:56.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:23:56.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:23:56.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:23:56.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:23:56.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:23:56.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:23:56.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:23:56.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:23:56.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:23:56.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:23:56.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-17T18:23:56.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:23:56.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:56.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:23:56.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21552 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-mapper 0 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T18:23:56.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2024-09-17T18:23:56.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:23:56.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:23:56.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:23:56.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:23:56.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:23:56.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:23:56.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:23:56.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:23:56.288 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:23:56.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:23:56.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:23:56.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:23:56.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:23:56.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:23:56.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:23:56.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:23:56.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:23:56.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T18:23:56.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2024-09-17T18:23:56.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:23:56.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:56.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21552 2024-09-17T18:23:56.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21552 2024-09-17T18:23:56.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:23:56.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:23:56.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T18:23:56.296 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:23:56.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:23:56.353 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-09-17T18:23:56.353 DEBUG:teuthology.orchestra.run.smithi105:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-09-17T18:23:56.410 INFO:tasks.workunit.client.0.smithi105.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:23:56.415 INFO:tasks.workunit.client.0.smithi105.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:23:56.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:23:56.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-recovery-scrub 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-09-17T18:23:56.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-09-17T18:23:56.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none ' 2024-09-17T18:23:56.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-17T18:23:56.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-17T18:23:56.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-17T18:23:56.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-09-17T18:23:56.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T18:23:56.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-09-17T18:23:56.420 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_recovery_scrub_2' 2024-09-17T18:23:56.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-17T18:23:56.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.37965 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-17T18:23:56.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:23:56.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:23:56.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:23:56.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:23:56.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:23:56.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:23:56.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:23:56.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:23:56.423 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:23:56.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:23:56.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:23:56.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:23:56.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:23:56.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:23:56.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:23:56.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:23:56.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:23:56.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:23:56.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-17T18:23:56.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:23:56.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:56.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:56.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37965 2024-09-17T18:23:56.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:23:56.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:23:56.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2024-09-17T18:23:56.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:23:56.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:56.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37965 2024-09-17T18:23:56.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:23:56.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T18:23:56.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T18:23:56.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2024-09-17T18:23:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:56.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:23:56.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-17T18:23:57.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:23:57.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:23:57.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:23:57.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:23:57.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:23:57.076 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:23:57.076 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37965/ceph-mon.a.asok 2024-09-17T18:23:57.077 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:23:57.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37965/ceph-mon.a.asok config get fsid 2024-09-17T18:23:57.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:23:57.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:23:57.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:23:57.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37965/ceph-mon.a.asok 2024-09-17T18:23:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:23:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37965/ceph-mon.a.asok config get mon_host 2024-09-17T18:23:57.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-09-17T18:23:57.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2024-09-17T18:23:57.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:23:57.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:23:57.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:23:57.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2024-09-17T18:23:57.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:23:58.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:23:58.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:58.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:58.189 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:58.189 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:58.189 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:58.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:23:58.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:23:58.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:23:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-09-17T18:23:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-09-17T18:23:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-09-17T18:23:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-09-17T18:23:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-09-17T18:23:58.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:23:58.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:23:58.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-17T18:23:58.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-09-17T18:23:58.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:23:58.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=26058680-47af-4149-b78a-1f8ea1a9cde4 2024-09-17T18:23:58.234 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 26058680-47af-4149-b78a-1f8ea1a9cde4 2024-09-17T18:23:58.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 26058680-47af-4149-b78a-1f8ea1a9cde4' 2024-09-17T18:23:58.235 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:23:58.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+yelm3a3UDhAAls7QB2BDuw455uz2K6OHmQ== 2024-09-17T18:23:58.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+yelm3a3UDhAAls7QB2BDuw455uz2K6OHmQ=="}' 2024-09-17T18:23:58.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 26058680-47af-4149-b78a-1f8ea1a9cde4 -i td/osd-recovery-scrub/0/new.json 2024-09-17T18:23:58.509 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:23:58.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-09-17T18:23:58.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQA+yelm3a3UDhAAls7QB2BDuw455uz2K6OHmQ== --osd-uuid 26058680-47af-4149-b78a-1f8ea1a9cde4 2024-09-17T18:23:58.568 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:58.562+0000 7f547fdba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:58.570 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:58.562+0000 7f547fdba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:58.572 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:58.566+0000 7f547fdba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:23:58.572 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:58.566+0000 7f547fdba7c0 -1 bdev(0x557275697c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:23:58.572 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:23:58.566+0000 7f547fdba7c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-09-17T18:24:04.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-09-17T18:24:04.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:24:04.750 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:24:04.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:24:04.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:24:05.148 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:24:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:24:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-17T18:24:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:24:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:24:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:24:05.181 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:05.174+0000 7faa228637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:05.202 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:05.198+0000 7faa228637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:05.203 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:05.198+0000 7faa228637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:24:05.452 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:24:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:24:05.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:06.265 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:06.258+0000 7faa228637c0 -1 Falling back to public interface 2024-09-17T18:24:06.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:06.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:06.782 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:24:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:24:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:24:07.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:08.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:08.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:08.123 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:24:08.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:24:08.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:08.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:24:08.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:08.975 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:08.970+0000 7faa228637c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:24:09.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:09.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:09.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:24:09.455 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:24:09.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:09.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:24:09.786 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3824179116,v1:127.0.0.1:6803/3824179116] [v2:127.0.0.1:6804/3824179116,v1:127.0.0.1:6805/3824179116] exists,up 26058680-47af-4149-b78a-1f8ea1a9cde4 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:24:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:24:09.791 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-17T18:24:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-09-17T18:24:09.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:24:09.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e83847e8-3c29-4506-9d79-85f49dbdd5ac 2024-09-17T18:24:09.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e83847e8-3c29-4506-9d79-85f49dbdd5ac' 2024-09-17T18:24:09.793 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 e83847e8-3c29-4506-9d79-85f49dbdd5ac 2024-09-17T18:24:09.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:24:09.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJyelm9EksMBAATRdar1eoBzHxsPvI1GrkUw== 2024-09-17T18:24:09.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJyelm9EksMBAATRdar1eoBzHxsPvI1GrkUw=="}' 2024-09-17T18:24:09.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e83847e8-3c29-4506-9d79-85f49dbdd5ac -i td/osd-recovery-scrub/1/new.json 2024-09-17T18:24:10.228 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:24:10.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-09-17T18:24:10.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBJyelm9EksMBAATRdar1eoBzHxsPvI1GrkUw== --osd-uuid e83847e8-3c29-4506-9d79-85f49dbdd5ac 2024-09-17T18:24:10.284 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:10.278+0000 7efc5ea587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:10.296 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:10.290+0000 7efc5ea587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:10.297 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:10.290+0000 7efc5ea587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:10.298 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:10.290+0000 7efc5ea587c0 -1 bdev(0x55cbe5923c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:24:10.298 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:10.290+0000 7efc5ea587c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-09-17T18:24:16.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-09-17T18:24:16.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:24:16.473 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:24:16.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:24:16.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:24:16.873 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:24:16.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:24:16.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:24:16.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-17T18:24:16.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:24:16.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:24:16.905 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:16.898+0000 7f21e279d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:16.907 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:16.902+0000 7f21e279d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:16.909 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:16.902+0000 7f21e279d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:17.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:24:17.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:24:17.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:24:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:24:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:24:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:24:17.225 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:24:17.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:17.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:24:17.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:17.925 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:17.922+0000 7f21e279d7c0 -1 Falling back to public interface 2024-09-17T18:24:18.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:18.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:18.619 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:24:18.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:24:18.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:18.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:24:18.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:19.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:19.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:19.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:24:19.944 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:24:19.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:19.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:24:20.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:20.687 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:20.682+0000 7f21e279d7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:24:21.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:21.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:21.264 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:24:21.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:24:21.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:21.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:24:21.600 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3938037560,v1:127.0.0.1:6811/3938037560] [v2:127.0.0.1:6812/3938037560,v1:127.0.0.1:6813/3938037560] exists,up e83847e8-3c29-4506-9d79-85f49dbdd5ac 2024-09-17T18:24:21.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:24:21.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:24:21.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:24:21.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:24:21.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-09-17T18:24:21.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-17T18:24:21.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:24:21.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:24:21.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:24:21.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-09-17T18:24:21.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-17T18:24:21.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:24:21.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:24:21.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:24:21.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-09-17T18:24:21.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-09-17T18:24:21.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:24:21.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:24:21.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-17T18:24:21.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:24:21.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:24:21.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:24:21.611 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:24:21.611 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:24:21.612 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:24:21.612 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:24:21.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:24:21.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:24:21.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:24:21.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:24:21.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-17T18:24:21.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-17T18:24:21.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:24:21.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:24:21.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:24:21.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:24:21.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:24:21.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-17T18:24:21.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-09-17T18:24:21.615 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:24:21.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1d89db72-4120-47bb-808d-25e272ff17bd 2024-09-17T18:24:21.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1d89db72-4120-47bb-808d-25e272ff17bd' 2024-09-17T18:24:21.616 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 1d89db72-4120-47bb-808d-25e272ff17bd 2024-09-17T18:24:21.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:24:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBVyelmZx9/JRAAlVUdAhn6K0qk5nO1mhFlWg== 2024-09-17T18:24:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBVyelmZx9/JRAAlVUdAhn6K0qk5nO1mhFlWg=="}' 2024-09-17T18:24:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1d89db72-4120-47bb-808d-25e272ff17bd -i td/osd-recovery-scrub/2/new.json 2024-09-17T18:24:22.028 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:24:22.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-09-17T18:24:22.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBVyelmZx9/JRAAlVUdAhn6K0qk5nO1mhFlWg== --osd-uuid 1d89db72-4120-47bb-808d-25e272ff17bd 2024-09-17T18:24:22.080 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:22.074+0000 7fee6621f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:22.105 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:22.102+0000 7fee6621f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:22.107 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:22.102+0000 7fee6621f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:22.107 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:22.102+0000 7fee6621f7c0 -1 bdev(0x5610f8cd1c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:24:22.107 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:22.102+0000 7fee6621f7c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-09-17T18:24:28.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-09-17T18:24:28.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:24:28.342 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:24:28.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:24:28.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:24:28.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:24:28.750 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:24:28.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:24:28.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-17T18:24:28.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:24:28.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:24:28.784 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:28.782+0000 7f86b04037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:28.817 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:28.814+0000 7f86b04037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:28.819 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:28.818+0000 7f86b04037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:29.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:24:29.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:24:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:24:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:24:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:24:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:29.090 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:24:29.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:24:29.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:29.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:29.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:30.081 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:30.078+0000 7f86b04037c0 -1 Falling back to public interface 2024-09-17T18:24:30.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:30.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:30.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:24:30.409 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:24:30.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:30.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:30.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:31.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:31.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:31.722 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:24:31.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:24:31.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:31.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:32.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:32.724 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:32.722+0000 7f86b04037c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:24:33.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:33.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:33.046 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:24:33.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:24:33.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:33.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:33.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:34.381 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:34.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:24:34.712 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/110611430,v1:127.0.0.1:6819/110611430] [v2:127.0.0.1:6820/110611430,v1:127.0.0.1:6821/110611430] exists,up 1d89db72-4120-47bb-808d-25e272ff17bd 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T18:24:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-09-17T18:24:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:24:34.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-09-17T18:24:34.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-09-17T18:24:34.720 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:24:34.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03949c28-3d0a-4632-b42b-6b30f8b95a4b 2024-09-17T18:24:34.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 03949c28-3d0a-4632-b42b-6b30f8b95a4b' 2024-09-17T18:24:34.720 INFO:tasks.workunit.client.0.smithi105.stdout:add osd3 03949c28-3d0a-4632-b42b-6b30f8b95a4b 2024-09-17T18:24:34.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:24:34.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBiyelmrsMhLBAAceTFKbLf5/g4D2W8bw4KdQ== 2024-09-17T18:24:34.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBiyelmrsMhLBAAceTFKbLf5/g4D2W8bw4KdQ=="}' 2024-09-17T18:24:34.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03949c28-3d0a-4632-b42b-6b30f8b95a4b -i td/osd-recovery-scrub/3/new.json 2024-09-17T18:24:35.106 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:24:35.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-09-17T18:24:35.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBiyelmrsMhLBAAceTFKbLf5/g4D2W8bw4KdQ== --osd-uuid 03949c28-3d0a-4632-b42b-6b30f8b95a4b 2024-09-17T18:24:35.160 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:35.158+0000 7f72173e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:35.173 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:35.170+0000 7f72173e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:35.174 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:35.174+0000 7f72173e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:35.175 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:35.174+0000 7f72173e57c0 -1 bdev(0x558b630f7c00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2024-09-17T18:24:35.175 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:35.174+0000 7f72173e57c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-09-17T18:24:41.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-09-17T18:24:41.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:24:41.084 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd3 key to auth repository 2024-09-17T18:24:41.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T18:24:41.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:24:41.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T18:24:41.502 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.3 2024-09-17T18:24:41.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=6d749d62-45a9-4c9f-b54a-2a4a2c4912b7 --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.37965/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-09-17T18:24:41.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:24:41.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:24:41.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:24:41.534 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:41.533+0000 7f054e8e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:41.562 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:41.561+0000 7f054e8e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:41.564 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:41.561+0000 7f054e8e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:41.851 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:24:41.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:24:41.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:41.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:42.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:42.829 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:42.825+0000 7f054e8e57c0 -1 Falling back to public interface 2024-09-17T18:24:43.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:43.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:43.190 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:24:43.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:24:43.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:43.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:43.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:44.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:44.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:44.519 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:24:44.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:24:44.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:44.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:44.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:45.787 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:24:45.785+0000 7f054e8e57c0 -1 osd.3 0 log_to_monitors true 2024-09-17T18:24:45.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:45.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:45.835 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:24:45.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:24:45.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:45.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:46.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:24:47.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:24:47.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:24:47.190 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:24:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:24:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:24:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:24:47.529 INFO:tasks.workunit.client.0.smithi105.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3832875105,v1:127.0.0.1:6827/3832875105] [v2:127.0.0.1:6828/3832875105,v1:127.0.0.1:6829/3832875105] exists,up 03949c28-3d0a-4632-b42b-6b30f8b95a4b 2024-09-17T18:24:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:24:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:24:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:24:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-09-17T18:24:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-17T18:24:47.960 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'test' created 2024-09-17T18:24:47.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:24:48.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-09-17T18:24:48.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:24:48.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:24:48.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:24:48.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:24:48.983 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:24:48.983 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:24:48.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:24:48.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:24:48.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:24:49.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:24:49.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:24:49.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:24:49.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:24:49.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:24:49.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:24:49.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:24:49.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836489 2024-09-17T18:24:49.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836489 2024-09-17T18:24:49.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489' 2024-09-17T18:24:49.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:24:49.706 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:24:49.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-17T18:24:49.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-17T18:24:49.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2024-09-17T18:24:49.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:24:49.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:24:49.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-17T18:24:49.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-17T18:24:49.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509445' 2024-09-17T18:24:49.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:24:49.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:24:50.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345922 2024-09-17T18:24:50.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345922 2024-09-17T18:24:50.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509445 3-85899345922' 2024-09-17T18:24:50.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:24:50.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836489 2024-09-17T18:24:50.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:24:50.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:24:50.134 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836489 2024-09-17T18:24:50.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:24:50.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836489 2024-09-17T18:24:50.135 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836489 2024-09-17T18:24:50.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-17T18:24:50.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:24:50.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836489 2024-09-17T18:24:50.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:24:51.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:24:51.464 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:24:51.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836489 2024-09-17T18:24:51.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:24:51.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-17T18:24:51.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:24:51.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:24:51.799 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-17T18:24:51.799 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:24:51.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-17T18:24:51.801 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672967 2024-09-17T18:24:51.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-17T18:24:51.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:24:52.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-17T18:24:52.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:24:52.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-17T18:24:52.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:24:52.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:24:52.130 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-17T18:24:52.130 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:24:52.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-17T18:24:52.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-17T18:24:52.131 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509445 2024-09-17T18:24:52.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:24:52.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-17T18:24:52.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:24:52.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345922 2024-09-17T18:24:52.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:24:52.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:24:52.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345922 2024-09-17T18:24:52.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:24:52.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345922 2024-09-17T18:24:52.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-09-17T18:24:52.469 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345922 2024-09-17T18:24:52.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:24:52.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345922 -lt 85899345922 2024-09-17T18:24:52.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:24:52.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:24:52.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:24:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:24:53.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:24:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:24:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:24:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:24:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:24:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:24:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:24:53.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:24:53.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:24:53.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:24:53.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:24:53.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:24:53.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:24:53.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:24:53.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-09-17T18:24:53.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-09-17T18:24:53.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-09-17T18:24:54.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-09-17T18:24:54.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-17T18:24:54.579 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:24:54.580 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-17T18:24:49.180900+0000 0'0 24:12 [1] 1 [1] 1 0'0 2024-09-17T18:24:47.905032+0000 0'0 2024-09-17T18:24:47.905032+0000 0 0 periodic scrub scheduled @ 2024-09-19T03:21:31.228504+0000 0 0 2024-09-17T18:24:54.580 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:24:54.580 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:24:54.580 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:24:54.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.37965 bs=1M count=50 2024-09-17T18:24:54.764 INFO:tasks.workunit.client.0.smithi105.stderr:50+0 records in 2024-09-17T18:24:54.764 INFO:tasks.workunit.client.0.smithi105.stderr:50+0 records out 2024-09-17T18:24:54.764 INFO:tasks.workunit.client.0.smithi105.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.168159 s, 312 MB/s 2024-09-17T18:24:54.765 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-09-17T18:24:54.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:24:54.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.37965 2024-09-17T18:24:56.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:24:56.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.37965 2024-09-17T18:24:57.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:24:57.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.37965 2024-09-17T18:24:59.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:24:59.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.37965 2024-09-17T18:25:00.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:00.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.37965 2024-09-17T18:25:02.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:02.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.37965 2024-09-17T18:25:03.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:03.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.37965 2024-09-17T18:25:05.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:05.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.37965 2024-09-17T18:25:06.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:06.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.37965 2024-09-17T18:25:08.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:08.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.37965 2024-09-17T18:25:09.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:09.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.37965 2024-09-17T18:25:11.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:11.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.37965 2024-09-17T18:25:12.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:12.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.37965 2024-09-17T18:25:14.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:14.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.37965 2024-09-17T18:25:15.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:15.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.37965 2024-09-17T18:25:17.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:17.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.37965 2024-09-17T18:25:18.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:18.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.37965 2024-09-17T18:25:20.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:20.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.37965 2024-09-17T18:25:21.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:21.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.37965 2024-09-17T18:25:23.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:23.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.37965 2024-09-17T18:25:24.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:24.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.37965 2024-09-17T18:25:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.37965 2024-09-17T18:25:27.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:27.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.37965 2024-09-17T18:25:29.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:29.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.37965 2024-09-17T18:25:31.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:31.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.37965 2024-09-17T18:25:32.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:32.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.37965 2024-09-17T18:25:34.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:34.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.37965 2024-09-17T18:25:35.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:35.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.37965 2024-09-17T18:25:37.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:37.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.37965 2024-09-17T18:25:38.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:38.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.37965 2024-09-17T18:25:40.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:40.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.37965 2024-09-17T18:25:41.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:41.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.37965 2024-09-17T18:25:43.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:43.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.37965 2024-09-17T18:25:45.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:45.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.37965 2024-09-17T18:25:46.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:46.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.37965 2024-09-17T18:25:48.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:48.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.37965 2024-09-17T18:25:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.37965 2024-09-17T18:25:51.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:51.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.37965 2024-09-17T18:25:52.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:52.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.37965 2024-09-17T18:25:54.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:54.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.37965 2024-09-17T18:25:55.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:55.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.37965 2024-09-17T18:25:57.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:57.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.37965 2024-09-17T18:25:58.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:25:58.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.37965 2024-09-17T18:26:00.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:00.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.37965 2024-09-17T18:26:01.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:01.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.37965 2024-09-17T18:26:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.37965 2024-09-17T18:26:04.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:04.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.37965 2024-09-17T18:26:06.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:06.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.37965 2024-09-17T18:26:07.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:07.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.37965 2024-09-17T18:26:09.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:09.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.37965 2024-09-17T18:26:10.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:10.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.37965 2024-09-17T18:26:12.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:12.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.37965 2024-09-17T18:26:13.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:13.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.37965 2024-09-17T18:26:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:15.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.37965 2024-09-17T18:26:16.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:16.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.37965 2024-09-17T18:26:18.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:18.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.37965 2024-09-17T18:26:19.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:19.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.37965 2024-09-17T18:26:21.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:21.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.37965 2024-09-17T18:26:23.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:23.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.37965 2024-09-17T18:26:24.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:24.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.37965 2024-09-17T18:26:26.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:26.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.37965 2024-09-17T18:26:27.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:27.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.37965 2024-09-17T18:26:28.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:28.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.37965 2024-09-17T18:26:30.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:30.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.37965 2024-09-17T18:26:32.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:32.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.37965 2024-09-17T18:26:33.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:33.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.37965 2024-09-17T18:26:34.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:34.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.37965 2024-09-17T18:26:36.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:36.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.37965 2024-09-17T18:26:37.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:37.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.37965 2024-09-17T18:26:39.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:39.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.37965 2024-09-17T18:26:40.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:40.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.37965 2024-09-17T18:26:42.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:42.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.37965 2024-09-17T18:26:44.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:44.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.37965 2024-09-17T18:26:45.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:45.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.37965 2024-09-17T18:26:47.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:47.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.37965 2024-09-17T18:26:49.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:49.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.37965 2024-09-17T18:26:50.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:50.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.37965 2024-09-17T18:26:52.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:52.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.37965 2024-09-17T18:26:53.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:53.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.37965 2024-09-17T18:26:55.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:55.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.37965 2024-09-17T18:26:56.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:56.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.37965 2024-09-17T18:26:58.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:26:58.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.37965 2024-09-17T18:27:00.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:00.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.37965 2024-09-17T18:27:01.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:01.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.37965 2024-09-17T18:27:03.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:03.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.37965 2024-09-17T18:27:04.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:04.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.37965 2024-09-17T18:27:06.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:06.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.37965 2024-09-17T18:27:07.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:07.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.37965 2024-09-17T18:27:09.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:09.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.37965 2024-09-17T18:27:10.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:10.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.37965 2024-09-17T18:27:12.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:12.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.37965 2024-09-17T18:27:13.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:13.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.37965 2024-09-17T18:27:15.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:15.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.37965 2024-09-17T18:27:16.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:16.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.37965 2024-09-17T18:27:18.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:18.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.37965 2024-09-17T18:27:20.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:20.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.37965 2024-09-17T18:27:21.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:21.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.37965 2024-09-17T18:27:23.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:23.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.37965 2024-09-17T18:27:24.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:24.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.37965 2024-09-17T18:27:26.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-09-17T18:27:26.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.37965 2024-09-17T18:27:28.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.37965 2024-09-17T18:27:28.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-09-17T18:27:28.497 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 1 size to 4 2024-09-17T18:27:28.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-09-17T18:27:28.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-09-17T18:27:28.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-17T18:27:28.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-17T18:27:28.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-17T18:27:28.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-17T18:27:28.860 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:27:28.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-17T18:27:30.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-09-17T18:27:30.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-09-17T18:27:30.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-09-17T18:27:30.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-17T18:27:30.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-17T18:27:30.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-17T18:27:30.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-17T18:27:31.198 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:27:31.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-09-17T18:27:33.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-09-17T18:27:33.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-09-17T18:27:33.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-09-17T18:27:33.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-09-17T18:27:33.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-09-17T18:27:33.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-09-17T18:27:33.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-09-17T18:27:33.695 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:27:33.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-09-17T18:27:33.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-09-17T18:27:33.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-09-17T18:27:34.033 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:27:34.033 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-09-17T18:27:30.519290+0000 24'1300 27:1324 [1,0,2,3] 1 [1,0] 1 0'0 2024-09-17T18:24:47.905032+0000 0'0 2024-09-17T18:24:47.905032+0000 0 0 no scrub is scheduled 0 0 2024-09-17T18:27:34.033 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:27:34.033 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:27:34.034 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:27:34.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-09-17T18:27:44.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-09-17T18:27:44.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:27:44.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:27:44.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:27:44.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:27:44.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:27:49.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:27:49.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-09-17T18:27:49.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2024-09-17T18:27:49.375 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-09-17T18:27:49.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-09-17T18:27:49.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-17T18:27:49.382 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:32.500+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.382 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:37.604+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.382 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:38.600+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.382 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:40.596+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.382 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:42.572+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.383 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:44.592+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.383 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:45.552+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.383 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:46.516+0000 7faa1b817640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-17T18:27:49.400 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:31.976+0000 7f21db751640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.400 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:33.976+0000 7f21db751640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.401 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:39.980+0000 7f21db751640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.401 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:42.980+0000 7f21db751640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-17T18:27:49.404 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:36.932+0000 7f86a93b7640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.404 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:39.956+0000 7f86a93b7640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.405 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:45.936+0000 7f86a93b7640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.405 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:47.860+0000 7f86a93b7640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-17T18:27:49.407 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:30.928+0000 7f0547899640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.407 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:40.852+0000 7f0547899640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.408 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:43.928+0000 7f0547899640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.408 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:27:46.968+0000 7f0547899640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2024-09-17T18:27:49.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-09-17T18:27:49.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-09-17T18:27:49.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-09-17T18:27:49.409 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-09-17T18:27:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-09-17T18:27:49.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2024-09-17T18:27:49.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-17T18:27:49.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-09-17T18:27:49.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-09-17T18:27:49.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2024-09-17T18:27:49.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-17T18:27:49.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-09-17T18:27:49.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-09-17T18:27:49.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2024-09-17T18:27:49.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-17T18:27:49.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-09-17T18:27:49.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-09-17T18:27:49.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-17T18:27:49.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2024-09-17T18:27:49.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-09-17T18:27:49.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-09-17T18:27:49.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:27:49.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:27:49.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:27:49.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:27:49.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:27:49.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:27:49.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:27:49.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:27:49.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:27:49.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:27:49.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:27:49.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:27:49.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:27:49.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:27:49.456 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:27:49.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:27:49.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-17T18:27:49.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:27:49.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:49.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:27:49.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37965 2024-09-17T18:27:49.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:27:49.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:27:49.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-09-17T18:27:49.483 INFO:tasks.workunit.client.0.smithi105.stdout:TEST PASSED 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: local poolname=test 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: return 0 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-recovery-scrub 0 2024-09-17T18:27:49.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:27:49.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:27:49.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:27:49.487 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:27:49.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:27:49.487 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:27:49.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:27:49.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:27:49.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:27:49.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:27:49.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:27:49.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:27:49.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T18:27:49.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2024-09-17T18:27:49.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:27:49.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:49.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37965 2024-09-17T18:27:49.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37965 2024-09-17T18:27:49.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:27:49.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:27:49.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T18:27:49.495 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:27:49.496 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:27:49.545 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-09-17T18:27:49.545 DEBUG:teuthology.orchestra.run.smithi105:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-09-17T18:27:49.598 INFO:tasks.workunit.client.0.smithi105.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:27:49.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:27:49.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:27:49.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:27:49.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-dump 2024-09-17T18:27:49.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-09-17T18:27:49.605 INFO:tasks.workunit.client.0.smithi105.stdout:This test is disabled 2024-09-17T18:27:49.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-09-17T18:27:49.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-17T18:27:49.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-dump 0 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:27:49.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:27:49.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:27:49.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:27:49.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:27:49.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:27:49.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:27:49.610 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:27:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:27:49.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:27:49.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:27:49.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-17T18:27:49.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2024-09-17T18:27:49.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:27:49.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:49.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46524 2024-09-17T18:27:49.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46524 2024-09-17T18:27:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:27:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:27:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-17T18:27:49.616 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-17T18:27:49.616 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T18:27:49.674 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-09-17T18:27:49.675 DEBUG:teuthology.orchestra.run.smithi105:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-09-17T18:27:49.725 INFO:tasks.workunit.client.0.smithi105.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-09-17T18:27:49.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ TIMEOUT=300 2024-09-17T18:27:49.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-17T18:27:49.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ MAX_TIMEOUT=15 2024-09-17T18:27:49.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ PG_NUM=4 2024-09-17T18:27:49.725 INFO:tasks.workunit.client.0.smithi105.stderr:++ TMPDIR=/tmp 2024-09-17T18:27:49.726 INFO:tasks.workunit.client.0.smithi105.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-17T18:27:49.726 INFO:tasks.workunit.client.0.smithi105.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-17T18:27:49.726 INFO:tasks.workunit.client.0.smithi105.stderr:+++ uname 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:++ '[' Linux = FreeBSD ']' 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:++ SED=sed 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:++ AWK=awk 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:+++ stty -a 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:+++ head -1 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-17T18:27:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-17T18:27:49.728 INFO:tasks.workunit.client.0.smithi105.stderr:++ termwidth= 2024-09-17T18:27:49.728 INFO:tasks.workunit.client.0.smithi105.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-17T18:27:49.728 INFO:tasks.workunit.client.0.smithi105.stderr:++ DIFFCOLOPTS='-y ' 2024-09-17T18:27:49.729 INFO:tasks.workunit.client.0.smithi105.stderr:++ KERNCORE=kernel.core_pattern 2024-09-17T18:27:49.729 INFO:tasks.workunit.client.0.smithi105.stderr:++ EXTRA_OPTS= 2024-09-17T18:27:49.732 INFO:tasks.workunit.client.0.smithi105.stderr:++ test '' = TESTS 2024-09-17T18:27:49.732 INFO:tasks.workunit.client.0.smithi105.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-09-17T18:27:49.732 INFO:tasks.workunit.client.0.smithi105.stderr:++ uname 2024-09-17T18:27:49.733 INFO:tasks.workunit.client.0.smithi105.stderr:+ '[' Linux = FreeBSD ']' 2024-09-17T18:27:49.733 INFO:tasks.workunit.client.0.smithi105.stderr:+ use_ec_overwrite=true 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr:+ getjson=no 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr:+ jqfilter='def walk(f): 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T18:27:49.734 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T18:27:49.735 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T18:27:49.735 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T18:27:49.735 INFO:tasks.workunit.client.0.smithi105.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T18:27:49.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ main osd-scrub-repair 2024-09-17T18:27:49.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ local dir=td/osd-scrub-repair 2024-09-17T18:27:49.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ shift 2024-09-17T18:27:49.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ shopt -s -o xtrace 2024-09-17T18:27:49.738 INFO:tasks.workunit.client.0.smithi105.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-17T18:27:49.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:27:49.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-17T18:27:49.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-scrub-repair 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-09-17T18:27:49.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-09-17T18:27:49.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none ' 2024-09-17T18:27:49.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-09-17T18:27:49.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-09-17T18:27:49.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-09-17T18:27:49.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-09-17T18:27:49.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-17T18:27:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-09-17T18:27:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_bluestore_basic 2024-09-17T18:27:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_bluestore_failed 2024-09-17T18:27:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_bluestore_scrub 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_bluestore_tag 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_erasure_coded_appends 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_and_repair_lrc_appends 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_and_repair_replicated 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_scrub_erasure_appends 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_scrub_replicated 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_corrupt_snapset_scrub_rep 2024-09-17T18:27:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_list_missing_erasure_coded_appends 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_list_missing_erasure_coded_overwrites 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_periodic_scrub_replicated 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_repair_stats 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_repair_stats_ec 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_request_scrub_priority 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_scrub_warning 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_skip_non_repair_during_recovery 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_unfound_erasure_coded_appends 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:TEST_unfound_erasure_coded_overwrites' 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:27:49.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:27:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:27:49.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:27:49.750 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:27:49.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:27:49.750 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:27:49.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:27:49.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:27:49.751 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:27:49.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:27:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:27:49.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:27:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:27:49.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:27:49.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:27:49.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:27:49.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:49.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:49.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:27:49.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:27:49.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:27:49.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:27:49.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:27:49.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:49.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:49.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-09-17T18:27:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:27:49.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-17T18:27:50.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:27:50.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:50.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:50.225 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:50.225 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:50.225 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:50.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:27:50.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-17T18:27:50.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:27:50.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:27:50.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:27:50.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:27:50.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:27:50.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:27:50.427 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:27:50.427 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:27:50.427 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:27:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:27:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:50.429 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:50.429 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:27:50.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:27:50.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:27:50.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:27:50.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:27:50.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:27:50.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:27:50.546 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:27:50.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-09-17T18:27:50.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:27:50.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:27:50.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:27:50.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:27:50.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:27:50.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:51.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:27:51.528 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:27:51.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:27:51.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-17T18:27:51.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:27:51.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:27:51.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:27:51.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:27:51.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:27:51.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:27:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:27:51.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:27:51.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:27:51.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:27:51.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-17T18:27:51.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:27:51.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:27:51.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=28648c7d-ea1c-4bf5-9351-363a6087d192 2024-09-17T18:27:51.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 28648c7d-ea1c-4bf5-9351-363a6087d192' 2024-09-17T18:27:51.569 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 28648c7d-ea1c-4bf5-9351-363a6087d192 2024-09-17T18:27:51.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:27:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAnyulm+8fqIhAAG64P3mRBJ1dCH5Y0MyDNMg== 2024-09-17T18:27:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAnyulm+8fqIhAAG64P3mRBJ1dCH5Y0MyDNMg=="}' 2024-09-17T18:27:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 28648c7d-ea1c-4bf5-9351-363a6087d192 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:27:51.821 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:27:51.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:27:51.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQAnyulm+8fqIhAAG64P3mRBJ1dCH5Y0MyDNMg== --osd-uuid 28648c7d-ea1c-4bf5-9351-363a6087d192 2024-09-17T18:27:51.875 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:51.872+0000 7fc7d59f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:51.877 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:51.876+0000 7fc7d59f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:51.878 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:51.876+0000 7fc7d59f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:51.878 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:51.876+0000 7fc7d59f47c0 -1 bdev(0x557373a59c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:27:51.879 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:51.876+0000 7fc7d59f47c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:27:58.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:27:58.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:27:58.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:27:58.106 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:27:58.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:27:58.504 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:27:58.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:27:58.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-17T18:27:58.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:27:58.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:27:58.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:27:58.539 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:58.536+0000 7f64f64387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:58.541 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:58.540+0000 7f64f64387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:58.542 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:58.540+0000 7f64f64387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:27:58.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:27:58.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:27:58.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:27:58.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:27:58.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:27:58.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:27:58.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:27:58.827 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:27:58.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:27:58.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:27:59.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:27:59.811 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:27:59.808+0000 7f64f64387c0 -1 Falling back to public interface 2024-09-17T18:28:00.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:00.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:00.144 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:28:00.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:28:00.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:00.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:00.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:01.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:01.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:01.478 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:28:01.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:28:01.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:01.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:01.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:02.496 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:02.492+0000 7f64f64387c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:28:02.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:02.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:02.794 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:28:02.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:28:02.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:02.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:03.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:04.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:04.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:04.127 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:28:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:28:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:04.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:04.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:05.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:05.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:05.448 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:28:05.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:28:05.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:05.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:05.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:06.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:06.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:06.771 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:28:06.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:28:06.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:06.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:07.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:08.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:08.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:08.093 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:28:08.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:28:08.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:08.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:08.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:09.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:09.421 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:28:09.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:09.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:28:09.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:09.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:09.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:10.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:10.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:10.743 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:28:10.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:28:10.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:10.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:11.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:12.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:12.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:12.060 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:28:12.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:28:12.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:12.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:12.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:13.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:13.452 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:28:13.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:13.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:28:13.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:13.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:13.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:14.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:14.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:14.789 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:28:14.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:28:14.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:14.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:15.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:16.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:16.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:16.326 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:28:16.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:28:16.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:16.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:17.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:18.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:18.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:18.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:28:18.298 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:28:18.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:18.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:19.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:20.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:20.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:20.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:28:20.229 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:28:20.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:20.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:20.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:21.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:21.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:21.951 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:28:21.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:28:21.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:21.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:22.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:23.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:23.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:23.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:28:23.657 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:28:23.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:23.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:24.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:25.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:25.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:25.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:28:25.134 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:28:25.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:25.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:28:25.462 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4217000588,v1:127.0.0.1:6803/4217000588] [v2:127.0.0.1:6804/4217000588,v1:127.0.0.1:6805/4217000588] exists,up 28648c7d-ea1c-4bf5-9351-363a6087d192 2024-09-17T18:28:25.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:28:25.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:28:25.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-09-17T18:28:25.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:28:25.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:28:25.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=661e3fc6-f8e6-43b8-b271-823dc87c8923 2024-09-17T18:28:25.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 661e3fc6-f8e6-43b8-b271-823dc87c8923' 2024-09-17T18:28:25.470 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 661e3fc6-f8e6-43b8-b271-823dc87c8923 2024-09-17T18:28:25.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:28:25.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJyulmG+1bHRAA2vpym/PwdlepqQr3wGSzbw== 2024-09-17T18:28:25.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJyulmG+1bHRAA2vpym/PwdlepqQr3wGSzbw=="}' 2024-09-17T18:28:25.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 661e3fc6-f8e6-43b8-b271-823dc87c8923 -i td/osd-scrub-repair/1/new.json 2024-09-17T18:28:25.852 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:28:25.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:28:25.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBJyulmG+1bHRAA2vpym/PwdlepqQr3wGSzbw== --osd-uuid 661e3fc6-f8e6-43b8-b271-823dc87c8923 2024-09-17T18:28:25.901 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:25.899+0000 7fb2aaa787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:25.937 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:25.935+0000 7fb2aaa787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:25.938 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:25.935+0000 7fb2aaa787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:25.939 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:25.935+0000 7fb2aaa787c0 -1 bdev(0x55c97a9b7c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:28:25.939 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:25.935+0000 7fb2aaa787c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:28:32.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:28:32.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:28:32.043 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:28:32.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:28:32.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:28:32.448 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:28:32.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-09-17T18:28:32.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:28:32.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:28:32.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:28:32.482 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:32.479+0000 7fa77f2f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:32.514 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:32.511+0000 7fa77f2f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:32.516 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:32.515+0000 7fa77f2f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:28:32.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:28:32.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:28:32.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:28:32.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:28:32.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:28:32.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:32.797 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:28:32.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:28:32.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:32.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:33.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:34.058 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:34.055+0000 7fa77f2f57c0 -1 Falling back to public interface 2024-09-17T18:28:34.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:34.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:34.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:28:34.116 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:28:34.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:34.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:34.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:35.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:35.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:35.465 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:28:35.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:28:35.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:35.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:35.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:36.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:36.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:36.789 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:28:36.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:28:36.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:36.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:37.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:37.236 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:28:37.235+0000 7fa77f2f57c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:28:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:38.121 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:28:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:28:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:38.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:39.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:39.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:39.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:28:39.489 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:28:39.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:39.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:39.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:40.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:40.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:40.813 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:28:40.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:28:40.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:40.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:41.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:42.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:42.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:42.189 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:28:42.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:28:42.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:42.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:42.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:43.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:43.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:43.528 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:28:43.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:28:43.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:43.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:43.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:44.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:44.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:44.899 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:28:44.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:28:44.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:44.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:45.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:46.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:46.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:46.236 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:28:46.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:28:46.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:46.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:46.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:47.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:47.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:47.576 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:28:47.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:28:47.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:47.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:47.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:48.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:48.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:48.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:28:48.939 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:28:48.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:48.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:49.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:50.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:50.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:50.501 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:28:50.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:28:50.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:50.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:51.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:52.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:52.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:52.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:28:52.393 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:28:52.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:52.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:52.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:53.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:53.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:53.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:28:53.758 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:28:53.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:53.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:54.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:55.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:55.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:55.509 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:28:55.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:28:55.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:55.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:56.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:57.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:57.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:57.253 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:28:57.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:28:57.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:57.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:28:58.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:28:58.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:28:58.912 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:28:58.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:28:58.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:28:58.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:28:59.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:00.632 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:29:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:29:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:00.963 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3313527280,v1:127.0.0.1:6811/3313527280] [v2:127.0.0.1:6812/3313527280,v1:127.0.0.1:6813/3313527280] exists,up 661e3fc6-f8e6-43b8-b271-823dc87c8923 2024-09-17T18:29:00.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:00.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:00.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:00.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-09-17T18:29:00.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T18:29:01.310 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T18:29:01.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T18:29:01.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T18:29:01.745 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T18:29:01.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:29:02.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T18:29:04.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-09-17T18:29:04.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:29:04.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:29:04.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:29:04.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:29:04.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:29:04.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:29:04.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:29:04.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:29:04.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:29:04.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:29:04.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:29:04.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:29:04.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:29:04.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:29:04.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:29:05.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:29:05.142 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T18:29:05.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:29:05.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:29:05.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:29:05.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-17T18:29:05.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-17T18:29:05.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-17T18:29:05.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:29:05.314 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:29:05.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672968 2024-09-17T18:29:05.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672968 2024-09-17T18:29:05.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492 1-42949672968' 2024-09-17T18:29:05.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:29:05.457 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-17T18:29:05.457 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:29:05.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:29:05.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-17T18:29:05.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:29:05.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-17T18:29:05.460 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836492 2024-09-17T18:29:05.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-17T18:29:05.461 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:29:05.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-17T18:29:05.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:29:06.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:29:06.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:29:07.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-17T18:29:07.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:29:07.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672968 2024-09-17T18:29:07.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:29:07.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:29:07.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672968 2024-09-17T18:29:07.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:29:07.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672968 2024-09-17T18:29:07.106 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672968 2024-09-17T18:29:07.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-17T18:29:07.107 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:29:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672968 2024-09-17T18:29:07.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:29:07.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:29:07.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:29:07.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:29:07.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:29:07.855 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:29:07.855 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:29:07.855 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:29:07.856 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:29:07.856 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:29:07.856 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:29:08.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:29:08.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:29:08.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:29:08.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:29:08.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:29:08.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:29:08.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:29:08.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-09-17T18:29:08.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T18:29:08.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-17T18:29:08.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-17T18:29:08.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T18:29:08.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T18:29:08.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T18:29:09.014 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T18:29:09.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T18:29:09.541 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T18:29:09.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T18:29:09.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T18:29:09.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:29:09.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-09-17T18:29:09.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-17T18:29:09.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:29:09.865 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-17T18:29:09.865 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-17T18:29:09.865 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:29:09.865 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-17T18:29:09.865 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:29:10.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:29:10.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-17T18:29:10.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:29:10.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-17T18:29:10.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T18:29:10.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-17T18:29:10.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-17T18:29:10.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:29:10.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:29:10.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:29:10.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:29:10.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:29:10.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:29:10.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:29:10.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:29:10.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:29:10.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:29:10.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:29:10.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:29:10.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:29:10.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:29:10.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:29:10.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:29:10.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:29:10.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:29:10.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:29:10.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:29:10.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:29:10.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:29:10.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:29:10.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-17T18:29:12.629 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:29:13.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:13.167 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:29:13.168 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:29:13.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:29:13.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:29:13.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:29:13.170 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:29:13.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:29:13.170 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:29:13.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:29:13.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:29:13.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:29:13.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:29:13.205 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:13.203+0000 7f0f740687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:13.208 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:13.207+0000 7f0f740687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:13.207+0000 7f0f740687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:13.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:13.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:14.218 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:14.215+0000 7f0f740687c0 -1 Falling back to public interface 2024-09-17T18:29:14.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:14.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:14.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:29:14.849 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:29:14.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:14.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:16.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:16.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:16.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:29:16.177 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:29:16.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:16.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:16.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:17.071 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:17.067+0000 7f0f740687c0 -1 osd.0 21 log_to_monitors true 2024-09-17T18:29:17.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:17.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:17.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:29:17.533 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:29:17.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:17.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2988011255,v1:127.0.0.1:6803/2988011255] [v2:127.0.0.1:6804/2988011255,v1:127.0.0.1:6805/2988011255] exists,up 28648c7d-ea1c-4bf5-9351-363a6087d192 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:29:17.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:29:17.956 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:29:17.956 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:29:17.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:29:17.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:29:17.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:29:18.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:29:18.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:29:18.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:29:18.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:29:18.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:29:18.065 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:29:18.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:29:18.447 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T18:29:18.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:29:18.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:29:18.448 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:29:18.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-17T18:29:18.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-17T18:29:18.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-17T18:29:18.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:29:18.598 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:29:18.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672972 2024-09-17T18:29:18.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672972 2024-09-17T18:29:18.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672972' 2024-09-17T18:29:18.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:29:18.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-17T18:29:18.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:29:18.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:29:18.762 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-17T18:29:18.762 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:29:18.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-17T18:29:18.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-17T18:29:18.764 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 111669149698 2024-09-17T18:29:18.764 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:29:19.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 111669149698 2024-09-17T18:29:19.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:29:20.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:29:20.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:29:20.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-17T18:29:20.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:29:20.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672972 2024-09-17T18:29:20.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:29:20.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:29:20.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672972 2024-09-17T18:29:20.435 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:29:20.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672972 2024-09-17T18:29:20.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-17T18:29:20.436 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672972 2024-09-17T18:29:20.437 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:29:20.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-17T18:29:20.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:29:20.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:29:20.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:29:21.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:29:21.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:29:21.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:29:21.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:29:21.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:29:21.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:29:21.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:29:21.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:29:21.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:29:21.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:29:21.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:29:21.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:29:21.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:29:21.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:29:21.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:29:21.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-17T18:29:21.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-17T18:29:21.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:29:21.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-17T18:29:21.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:29:22.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-17T18:29:22.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-17T18:29:22.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-17T18:29:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-17T18:29:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T18:29:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:29:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:29:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T18:29:22.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:22.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-17T18:29:22.811 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-17T18:29:22.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:22.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-17T18:29:22.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:22.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:29:22.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:29:22.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:29:22.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T18:29:22.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T18:29:22.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:29:22.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:29:22.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T18:29:23.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:29:01.698109+0000 '>' 2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:23.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:29:24.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:29:24.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:29:24.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T18:29:24.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T18:29:24.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:29:24.134 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:29:24.134 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T18:29:24.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:29:01.698109+0000 '>' 2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:24.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:29:25.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:29:25.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:29:25.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T18:29:25.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T18:29:25.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:29:25.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:29:25.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T18:29:25.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:29:01.698109+0000 '>' 2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:25.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:29:26.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:29:26.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:29:26.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T18:29:26.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T18:29:26.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:29:26.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:29:26.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T18:29:27.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:29:23.496239+0000 '>' 2024-09-17T18:29:01.698109+0000 2024-09-17T18:29:27.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:29:27.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:29:29.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:29:29.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:29:29.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:29:29.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:29:29.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:29:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:29:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:29:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:29:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:29:29.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:29:29.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:29:29.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:29:29.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:29:29.926 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T18:29:29.927 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:29:29.927 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:29:30.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:30.219 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:29:30.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:29:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:29:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:29:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:29:30.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:29:30.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:29:30.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:29:30.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:29:30.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:29:30.254 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:30.251+0000 7f45364267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:30.274 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:30.271+0000 7f45364267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:30.276 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:30.275+0000 7f45364267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:29:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:29:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:29:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:29:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:29:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:30.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:29:30.548 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:29:30.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:30.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:30.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:31.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:31.331+0000 7f45364267c0 -1 Falling back to public interface 2024-09-17T18:29:31.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:31.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:31.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:29:31.896 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:29:31.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:31.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:32.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:33.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:33.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:33.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:29:33.225 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:29:33.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:33.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:33.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:34.508 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:34.507+0000 7f45364267c0 -1 osd.1 28 log_to_monitors true 2024-09-17T18:29:34.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:34.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:34.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:29:34.565 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:29:34.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:34.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:34.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:29:35.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:29:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:29:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:29:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:29:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:29:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:29:36.310 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/651248071,v1:127.0.0.1:6811/651248071] [v2:127.0.0.1:6812/651248071,v1:127.0.0.1:6813/651248071] exists,up 661e3fc6-f8e6-43b8-b271-823dc87c8923 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:29:36.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:29:36.312 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:29:36.312 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:29:36.312 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:29:36.312 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:29:36.312 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:29:36.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:29:36.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:29:36.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:29:36.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:29:36.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:29:36.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:29:36.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:29:36.752 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T18:29:36.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:29:36.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:29:36.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:29:36.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-17T18:29:36.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-17T18:29:36.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-17T18:29:36.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:29:36.908 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:29:37.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-17T18:29:37.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-17T18:29:37.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-17T18:29:37.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:29:37.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-17T18:29:37.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:29:37.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:29:37.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-17T18:29:37.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:29:37.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-17T18:29:37.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-17T18:29:37.054 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 111669149702 2024-09-17T18:29:37.055 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:29:37.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-17T18:29:37.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:29:38.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:29:38.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:29:38.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149703 -lt 111669149702 2024-09-17T18:29:38.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:29:38.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-17T18:29:38.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:29:38.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:29:38.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-17T18:29:38.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:29:38.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-17T18:29:38.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-17T18:29:38.697 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 137438953474 2024-09-17T18:29:38.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:29:39.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-17T18:29:39.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:29:39.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:29:39.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:29:39.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:29:39.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:29:39.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:29:39.768 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:29:39.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:29:39.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:29:40.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:29:40.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:29:40.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:29:40.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:29:40.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:29:40.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:29:40.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:29:40.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:29:40.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:29:40.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:29:40.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:29:40.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:29:40.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:29:40.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:29:40.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:29:40.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:29:40.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:29:40.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:29:40.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:29:40.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:29:40.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:29:40.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:29:40.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:29:40.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:29:40.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:29:40.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:29:40.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:29:40.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:29:40.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:29:40.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:40.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:29:40.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:29:40.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:29:40.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:29:40.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:29:40.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:29:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:29:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:29:40.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:29:40.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:29:40.398 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:29:40.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:29:40.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:29:40.399 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:29:40.400 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:29:40.400 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:29:40.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:29:40.401 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:29:40.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:29:40.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:29:40.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:29:40.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:29:40.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:40.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:29:40.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:29:40.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:29:40.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:29:40.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:29:40.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:29:40.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:29:40.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:29:40.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-09-17T18:29:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:234: standard_scrub_cluster: local poolname=testpool 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: args['pool_name']=testpool 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:237: standard_scrub_cluster: local debug_msg=dbg 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:240: standard_scrub_cluster: local saved_echo_flag=x 2024-09-17T18:29:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: set +x 2024-09-17T18:29:42.182 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 9ffbe981-d59a-40c9-a3ba-4e08727cd95e 2024-09-17T18:29:42.434 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:29:42.485 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:42.483+0000 7f00876b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:42.487 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:42.487+0000 7f00876b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:42.488 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:42.487+0000 7f00876b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:42.489 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:42.487+0000 7f00876b87c0 -1 bdev(0x560f90447c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:29:42.489 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:42.487+0000 7f00876b87c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:29:48.654 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:29:49.074 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:29:49.114 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:49.111+0000 7fd7989da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:49.140 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:49.139+0000 7fd7989da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:49.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:49.139+0000 7fd7989da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:29:49.438 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:29:50.702 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:50.699+0000 7fd7989da7c0 -1 Falling back to public interface 2024-09-17T18:29:50.764 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:29:52.105 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:29:53.436 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:29:53.434+0000 7fd7989da7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:29:53.443 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:29:54.876 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:29:56.196 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:29:57.524 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:29:58.858 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:30:00.209 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:30:01.527 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:30:02.859 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:30:04.201 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:30:05.533 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:30:07.232 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:30:09.056 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:30:10.880 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:30:12.314 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:30:13.849 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:30:15.399 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:30:15.734 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/274121923,v1:127.0.0.1:6803/274121923] [v2:127.0.0.1:6804/274121923,v1:127.0.0.1:6805/274121923] exists,up 9ffbe981-d59a-40c9-a3ba-4e08727cd95e 2024-09-17T18:30:15.741 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 a0a9ea03-373c-4a4b-9c38-5e065d4c4712 2024-09-17T18:30:16.112 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:30:16.174 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:16.170+0000 7f0c37b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:16.197 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:16.194+0000 7f0c37b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:16.199 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:16.198+0000 7f0c37b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:16.199 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:16.198+0000 7f0c37b4b7c0 -1 bdev(0x564db056dc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:30:16.199 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:16.198+0000 7f0c37b4b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:30:22.789 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:30:23.201 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:30:23.242 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:23.238+0000 7fb78ce077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:23.268 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:23.266+0000 7fb78ce077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:23.269 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:23.266+0000 7fb78ce077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:23.532 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:30:24.342 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:24.338+0000 7fb78ce077c0 -1 Falling back to public interface 2024-09-17T18:30:24.892 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:30:26.204 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:30:27.237 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:27.234+0000 7fb78ce077c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:30:27.556 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:30:29.016 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:30:30.507 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:30:31.841 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:30:33.183 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:30:34.516 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:30:35.833 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:30:37.151 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:30:38.500 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:30:39.826 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:30:41.479 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:30:43.136 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:30:44.881 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:30:46.252 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:30:47.966 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:30:49.610 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:30:49.939 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/632584268,v1:127.0.0.1:6811/632584268] [v2:127.0.0.1:6812/632584268,v1:127.0.0.1:6813/632584268] exists,up a0a9ea03-373c-4a4b-9c38-5e065d4c4712 2024-09-17T18:30:49.945 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 5848c177-cedb-4c6d-973c-f730b65263b3 2024-09-17T18:30:50.345 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:30:50.407 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:50.406+0000 7f88e26797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:50.421 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:50.418+0000 7f88e26797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:50.422 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:50.422+0000 7f88e26797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:50.423 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:50.422+0000 7f88e26797c0 -1 bdev(0x559a8e96fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:30:50.423 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:50.422+0000 7f88e26797c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:30:56.640 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:30:57.046 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:30:57.085 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:57.082+0000 7fdb576b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:57.117 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:57.114+0000 7fdb576b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:57.119 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:57.118+0000 7fdb576b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:30:57.390 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:30:58.629 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:30:58.626+0000 7fdb576b97c0 -1 Falling back to public interface 2024-09-17T18:30:58.696 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:31:00.009 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:31:01.327 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:31:01.462 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:31:01.462+0000 7fdb576b97c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:31:02.676 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:31:03.993 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:31:05.310 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:31:06.638 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:31:07.972 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:31:09.307 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:31:10.632 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:31:11.970 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:31:13.304 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:31:14.621 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:31:16.130 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:31:17.872 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:31:19.545 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:31:21.213 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:31:22.675 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:31:24.284 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:31:24.627 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/25723134,v1:127.0.0.1:6819/25723134] [v2:127.0.0.1:6820/25723134,v1:127.0.0.1:6821/25723134] exists,up 5848c177-cedb-4c6d-973c-f730b65263b3 2024-09-17T18:31:25.112 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'testpool' created 2024-09-17T18:31:27.074 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836500 2024-09-17T18:31:27.400 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672973 2024-09-17T18:31:29.049 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509445 2024-09-17T18:31:31.806 INFO:tasks.workunit.client.0.smithi105.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-09-17T18:31:31.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-09-17T18:31:31.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-09-17T18:31:31.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-09-17T18:31:32.227 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 1 size to 2 2024-09-17T18:31:32.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-17T18:31:32.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:31:32.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:31:32.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:31:32.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:31:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:31:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:31:32.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:31:32.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:31:32.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:31:32.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:31:32.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:31:32.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:31:32.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:31:32.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:31:32.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:31:32.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:31:32.819 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:31:32.819 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:31:32.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:31:32.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:32.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:31:32.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-17T18:31:32.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-17T18:31:32.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-17T18:31:32.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:32.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:31:33.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-17T18:31:33.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-17T18:31:33.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672975' 2024-09-17T18:31:33.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:33.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:31:33.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-17T18:31:33.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-17T18:31:33.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672975 2-64424509447' 2024-09-17T18:31:33.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:33.256 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-17T18:31:33.256 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:33.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:31:33.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-17T18:31:33.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:33.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-17T18:31:33.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-17T18:31:33.259 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836502 2024-09-17T18:31:33.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:31:33.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836502 2024-09-17T18:31:33.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:31:34.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:31:34.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:31:35.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-17T18:31:35.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:35.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-17T18:31:35.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:35.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:31:35.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-17T18:31:35.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:35.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-17T18:31:35.085 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672975 2024-09-17T18:31:35.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-17T18:31:35.085 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:31:35.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672975 2024-09-17T18:31:35.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:35.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-17T18:31:35.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:35.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:31:35.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-17T18:31:35.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:35.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-17T18:31:35.432 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509447 2024-09-17T18:31:35.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-17T18:31:35.433 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:31:35.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-17T18:31:35.768 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:31:35.768 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:31:35.768 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:31:36.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:31:36.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:31:36.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:31:36.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:31:36.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:31:36.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:31:36.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:31:36.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:31:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:31:36.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:31:36.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:31:36.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:31:36.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:31:36.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:31:36.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:31:36.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-09-17T18:31:36.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-09-17T18:31:36.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:31:37.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-17T18:31:37.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:31:37.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:31:37.024 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:31:37.024 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:31:37.025 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:31:37.025 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:31:37.025 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:31:37.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:31:37.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:31:37.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:31:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:31:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:31:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:31:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:31:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:31:37.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:31:37.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:31:38.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:31:38.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:31:38.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:31:38.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:31:38.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:31:38.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:31:38.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:31:38.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-17T18:31:39.900 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-17T18:31:40.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T18:31:40.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:31:40.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:31:40.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:31:40.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:31:40.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:31:40.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:31:40.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:31:40.440 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:31:40.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:31:40.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:31:40.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:31:40.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:31:40.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:31:40.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:31:40.474 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:31:40.473+0000 7ff73bc4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:40.476 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:31:40.477+0000 7ff73bc4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:40.477 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:31:40.477+0000 7ff73bc4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:31:40.770 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:31:40.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:40.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:41.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:42.021 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:31:42.021+0000 7ff73bc4e7c0 -1 Falling back to public interface 2024-09-17T18:31:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:31:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:31:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:42.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:42.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:43.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:43.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:43.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:31:43.463 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:31:43.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:43.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:43.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:44.722 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:31:44.725+0000 7ff73bc4e7c0 -1 osd.0 21 log_to_monitors true 2024-09-17T18:31:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:31:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:31:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:45.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:31:46.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:31:46.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:31:46.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:31:46.121 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:31:46.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:31:46.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:31:46.434 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1263924936,v1:127.0.0.1:6803/1263924936] [v2:127.0.0.1:6804/1263924936,v1:127.0.0.1:6805/1263924936] exists,up 9ffbe981-d59a-40c9-a3ba-4e08727cd95e 2024-09-17T18:31:46.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:31:46.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:31:46.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:31:46.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:31:46.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:31:46.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:31:46.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:31:46.435 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:31:46.435 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:31:46.435 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:31:46.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:31:46.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:31:46.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:31:46.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:31:46.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:31:46.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:31:46.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:31:46.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:31:46.539 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:31:46.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:31:46.856 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:31:46.856 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:31:46.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:31:46.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:46.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:31:47.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-17T18:31:47.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-17T18:31:47.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-17T18:31:47.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:47.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:31:47.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-17T18:31:47.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-17T18:31:47.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672979' 2024-09-17T18:31:47.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:47.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:31:47.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-17T18:31:47.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-17T18:31:47.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672979 2-64424509451' 2024-09-17T18:31:47.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:47.318 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-17T18:31:47.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:47.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:31:47.320 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-17T18:31:47.320 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:47.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-17T18:31:47.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-17T18:31:47.321 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 107374182402 2024-09-17T18:31:47.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:31:47.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-17T18:31:47.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:31:48.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:31:48.619 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:31:48.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-17T18:31:48.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:48.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-17T18:31:48.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:48.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:31:48.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-17T18:31:48.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-17T18:31:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-17T18:31:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672979 2024-09-17T18:31:48.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:31:49.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-17T18:31:49.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:49.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:49.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-17T18:31:49.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:31:49.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-17T18:31:49.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:49.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-17T18:31:49.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-17T18:31:49.292 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509451 2024-09-17T18:31:49.293 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:31:49.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-17T18:31:49.626 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:31:49.627 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:31:49.627 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:31:50.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:31:50.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:31:50.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:31:50.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:31:50.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:31:50.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:31:50.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:31:50.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:31:50.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:31:50.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:31:50.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:31:50.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:31:50.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:31:50.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:31:50.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:31:50.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-17T18:31:50.967 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0: { 2024-09-17T18:31:50.967 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-17T18:31:50.968 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:31:50.980 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1: { 2024-09-17T18:31:50.980 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-17T18:31:50.980 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:31:50.993 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2: { 2024-09-17T18:31:50.993 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-17T18:31:50.993 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:31:51.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-09-17T18:31:51.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-17T18:31:51.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:31:51.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-17T18:31:51.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:31:51.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-09-17T18:31:51.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-09-17T18:31:51.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:31:51.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:31:51.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:31:51.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:31:51.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-09-17T18:31:51.729 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-09-17T18:31:51.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:31:51.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:31:51.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:31:51.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:31:52.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-09-17T18:31:25.058272+0000 2024-09-17T18:31:52.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-09-17T18:31:52.209 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:31:52.209 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": true, 2024-09-17T18:31:52.209 INFO:tasks.workunit.client.0.smithi105.stdout: "must": false, 2024-09-17T18:31:52.209 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "2024-09-03T18:30:12.212921+0000" 2024-09-17T18:31:52.209 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:31:52.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-09-17T18:31:25.058272+0000 2024-09-17T18:31:52.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-17T18:31:52.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:31:25.058272+0000 2024-09-17T18:31:52.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:31:52.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:31:52.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:31:52.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:31:52.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:31:52.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:31:52.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:31:52.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:31:52.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:31:25.058272+0000 '>' 2024-09-17T18:31:25.058272+0000 2024-09-17T18:31:52.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:31:53.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:31:53.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:31:53.539 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:31:53.539 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:31:53.539 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:31:53.539 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:31:53.540 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:31:53.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:31:25.058272+0000 '>' 2024-09-17T18:31:25.058272+0000 2024-09-17T18:31:53.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:31:54.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:31:54.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:31:54.859 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:31:54.859 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:31:54.859 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:31:54.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:31:54.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:31:53.039335+0000 '>' 2024-09-17T18:31:25.058272+0000 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:31:55.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:31:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:31:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:31:55.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:31:55.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:31:55.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:31:55.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:31:55.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:31:55.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:31:55.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:31:55.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:31:55.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:31:55.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:31:55.591 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:31:55.591 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:31:55.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:31:55.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:55.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:31:55.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182405 2024-09-17T18:31:55.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182405 2024-09-17T18:31:55.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405' 2024-09-17T18:31:55.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:55.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:31:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-17T18:31:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-17T18:31:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-42949672982' 2024-09-17T18:31:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:31:55.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:31:56.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-17T18:31:56.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-17T18:31:56.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-42949672982 2-64424509454' 2024-09-17T18:31:56.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:56.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182405 2024-09-17T18:31:56.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:56.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:31:56.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182405 2024-09-17T18:31:56.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:56.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182405 2024-09-17T18:31:56.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-17T18:31:56.042 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 107374182405 2024-09-17T18:31:56.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:31:56.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182403 -lt 107374182405 2024-09-17T18:31:56.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:31:57.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:31:57.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:31:57.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-17T18:31:57.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:57.680 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-17T18:31:57.680 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:57.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:31:57.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-17T18:31:57.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:57.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-17T18:31:57.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-17T18:31:57.684 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672982 2024-09-17T18:31:57.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:31:58.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-17T18:31:58.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:31:58.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:31:58.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-17T18:31:58.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:31:58.025 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-17T18:31:58.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:31:58.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-17T18:31:58.027 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509454 2024-09-17T18:31:58.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-17T18:31:58.027 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:31:58.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-17T18:31:58.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:31:58.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:31:58.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:31:58.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:31:59.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:31:59.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:31:59.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:31:59.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:31:59.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:31:59.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:31:59.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:31:59.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-09-17T18:31:59.884 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:31:59.884 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-17T18:31:53.215464+0000 21'1 26:63 [1,0] 1 [1,0] 1 21'1 2024-09-17T18:31:53.039335+0000 21'1 2024-09-17T18:31:53.039335+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:31:53.039335+0000 1 0 2024-09-17T18:31:59.884 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:31:59.884 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:31:59.885 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:31:59.902 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-17T18:31:59.902 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:31:59.902 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:31:59.903 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:31:59.903 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:31:59.903 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:31:59.903 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:31:59.903 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:32:00.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:32:00.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:32:00.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:32:00.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:32:00.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:32:00.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:32:00.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:32:00.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:32:00.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:32:00.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:32:00.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:32:00.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:32:00.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:32:00.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:32:00.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:32:00.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:32:00.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:32:00.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-17T18:32:01.459 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:32:01.459 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:32:01.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T18:32:01.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:32:01.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:32:01.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:32:01.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:32:01.746 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:32:01.746 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:01.746 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:01.746 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:32:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:32:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:32:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:32:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:32:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:32:01.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:32:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:32:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:32:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:32:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:32:01.782 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:01.785+0000 7efc14bf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:01.785 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:01.785+0000 7efc14bf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:01.786 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:01.789+0000 7efc14bf87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:02.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:32:02.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:32:02.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:32:02.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:32:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:32:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:32:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:32:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:02.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:02.885 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:02.885+0000 7efc14bf87c0 -1 Falling back to public interface 2024-09-17T18:32:03.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:03.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:03.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:32:03.437 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:32:03.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:03.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:03.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:04.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:04.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:04.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:32:04.767 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:32:04.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:04.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:05.986 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:05.989+0000 7efc14bf87c0 -1 osd.0 26 log_to_monitors true 2024-09-17T18:32:06.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:06.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:06.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:32:06.109 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:32:06.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:06.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2146607498,v1:127.0.0.1:6803/2146607498] [v2:127.0.0.1:6804/2146607498,v1:127.0.0.1:6805/2146607498] exists,up 9ffbe981-d59a-40c9-a3ba-4e08727cd95e 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:32:06.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:32:06.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:32:06.504 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:32:06.504 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:32:06.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:32:06.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:32:06.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:32:06.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:32:06.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:32:06.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:32:06.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:32:06.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:32:06.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:32:06.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:32:06.945 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:32:06.945 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:32:06.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:32:06.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:06.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:32:07.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-17T18:32:07.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-17T18:32:07.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-17T18:32:07.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:07.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:32:07.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-17T18:32:07.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-17T18:32:07.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-09-17T18:32:07.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:07.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:32:07.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-17T18:32:07.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-17T18:32:07.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509457' 2024-09-17T18:32:07.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:07.479 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-17T18:32:07.479 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:07.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:32:07.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-17T18:32:07.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:07.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-17T18:32:07.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-17T18:32:07.481 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 128849018882 2024-09-17T18:32:07.482 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:32:07.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-17T18:32:07.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:32:08.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:32:08.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:32:09.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-17T18:32:09.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:09.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-17T18:32:09.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:09.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:32:09.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-17T18:32:09.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:09.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-17T18:32:09.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-17T18:32:09.154 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672985 2024-09-17T18:32:09.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:32:09.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-17T18:32:09.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:09.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-17T18:32:09.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:09.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:32:09.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-17T18:32:09.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:09.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-17T18:32:09.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-17T18:32:09.488 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509457 2024-09-17T18:32:09.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:32:09.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-17T18:32:09.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:32:09.799 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:09.799 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:10.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:32:10.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:32:10.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:32:10.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:32:10.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:32:10.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:32:10.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:32:10.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:32:10.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:32:10.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:32:10.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:10.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:10.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:32:10.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:32:10.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:32:10.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:32:10.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:32:10.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:32:11.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:32:11.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:32:11.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:32:11.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:32:11.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:32:11.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:32:11.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:32:11.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:32:11.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:32:11.354 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:32:11.354 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:11.354 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:32:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:32:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:32:11.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-09-17T18:32:11.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:32:11.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:32:11.743 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:32:11.743 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:32:11.743 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:32:11.744 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:32:11.744 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:32:12.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:32:12.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:32:12.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:32:12.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:32:12.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:32:12.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:32:12.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:32:12.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:32:12.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:32:12.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:32:12.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:32:12.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:32:12.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:32:12.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:32:12.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:32:12.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:32:12.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:32:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:32:13.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:32:13.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:32:13.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:13.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:13.511 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:13.511 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:32:13.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:32:13.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:32:13.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:32:13.513 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:32:13.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:32:13.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:32:13.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:32:13.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:32:13.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:32:13.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:32:13.547 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:13.548+0000 7f330e9cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:13.550 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:13.552+0000 7f330e9cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:13.552 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:13.552+0000 7f330e9cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:13.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:32:13.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:32:13.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:13.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:14.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:14.825 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:14.824+0000 7f330e9cb7c0 -1 Falling back to public interface 2024-09-17T18:32:15.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:15.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:15.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:32:15.158 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:32:15.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:15.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:15.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:32:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:32:16.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:16.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:16.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:17.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:17.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:17.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:32:17.804 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:32:17.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:17.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:18.039 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:18.040+0000 7f330e9cb7c0 -1 osd.0 31 log_to_monitors true 2024-09-17T18:32:18.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:19.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:19.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:19.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:32:19.144 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:32:19.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:19.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/939345350,v1:127.0.0.1:6803/939345350] [v2:127.0.0.1:6804/939345350,v1:127.0.0.1:6805/939345350] exists,up 9ffbe981-d59a-40c9-a3ba-4e08727cd95e 2024-09-17T18:32:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:32:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:32:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:32:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:32:19.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:32:19.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:32:19.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:32:19.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:32:19.484 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:32:19.484 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:32:19.485 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:32:19.485 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:32:19.485 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:32:19.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:32:19.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:32:19.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:32:19.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:32:19.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:32:19.587 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:32:19.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:32:19.936 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:32:19.936 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:32:19.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:32:19.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:19.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:32:20.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-17T18:32:20.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-17T18:32:20.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-17T18:32:20.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:20.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:32:20.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672989 2024-09-17T18:32:20.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672989 2024-09-17T18:32:20.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672989' 2024-09-17T18:32:20.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:32:20.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:32:20.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-17T18:32:20.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-17T18:32:20.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672989 2-64424509461' 2024-09-17T18:32:20.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:20.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:20.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-17T18:32:20.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:32:20.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-17T18:32:20.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:20.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-17T18:32:20.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-17T18:32:20.377 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 150323855362 2024-09-17T18:32:20.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:32:20.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 150323855362 2024-09-17T18:32:20.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:32:21.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:32:21.707 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:32:22.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-17T18:32:22.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:22.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:22.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672989 2024-09-17T18:32:22.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:32:22.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672989 2024-09-17T18:32:22.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:22.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672989 2024-09-17T18:32:22.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-09-17T18:32:22.035 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672989 2024-09-17T18:32:22.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:32:22.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672989 -lt 42949672989 2024-09-17T18:32:22.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:32:22.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-17T18:32:22.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:32:22.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:32:22.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-17T18:32:22.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:32:22.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-17T18:32:22.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-17T18:32:22.376 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509461 2024-09-17T18:32:22.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:32:22.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-17T18:32:22.736 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:32:22.737 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:22.737 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:23.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:32:23.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:32:23.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:32:23.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:32:23.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:32:23.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:32:23.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:32:23.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:32:23.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:32:23.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:32:23.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:32:23.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:32:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:32:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:32:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:32:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:32:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-17T18:32:23.875 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:31:53.037+0000 7fb770d91640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:32:23.875 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:31:53.037+0000 7fb770d91640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:32:23.875 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:31:53.037+0000 7fb770d91640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-17T18:32:23.875 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:31:53.037+0000 7fb770d91640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-17T18:32:23.876 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:31:53.037+0000 7fb770d91640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=16/16 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:32:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:32:24.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:32:24.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:32:24.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:32:24.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:32:24.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:32:24.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:32:24.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:32:24.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:24.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:32:24.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:32:24.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:24.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:32:24.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:32:24.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:32:24.215 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:32:24.215 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:24.215 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:24.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:32:24.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:32:24.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:32:24.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:32:24.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:32:24.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:32:24.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:32:24.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:32:24.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:32:24.220 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:32:24.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:32:24.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:32:24.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:32:24.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:32:24.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:32:24.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:24.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:32:24.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:32:24.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:32:24.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:32:24.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:32:24.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:32:24.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:32:24.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:24.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:24.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:32:24.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:32:24.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:32:24.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:32:24.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:32:24.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:24.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:24.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:32:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:32:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:32:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:32:24.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:32:24.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:24.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:24.698 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:24.698 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:24.698 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:24.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:24.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:32:24.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:32:24.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:32:24.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:32:24.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:32:24.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:32:24.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:32:24.917 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:32:24.917 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:32:24.917 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:32:24.917 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:32:24.918 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:24.918 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:24.918 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:32:24.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:32:24.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:32:25.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:32:25.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:32:25.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:32:25.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:32:25.034 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:32:25.034 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:32:25.034 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:32:25.034 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:32:25.035 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:32:25.035 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:25.035 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:25.035 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:32:25.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:32:25.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:32:25.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-09-17T18:32:25.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:32:25.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:32:25.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:32:25.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:32:25.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:32:25.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:32:26.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:32:26.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:26.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:26.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:26.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:32:26.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:32:26.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:32:26.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:32:26.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:26.043 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:32:26.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:32:26.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:32:26.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c44bc72c-483a-4594-9a03-3f9f0e69fb58' 2024-09-17T18:32:26.045 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:32:26.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:32:26.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA6y+lmOBjXAxAAH99rCH3EVdvdjP0h57xfWA== 2024-09-17T18:32:26.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA6y+lmOBjXAxAAH99rCH3EVdvdjP0h57xfWA=="}' 2024-09-17T18:32:26.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c44bc72c-483a-4594-9a03-3f9f0e69fb58 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:32:26.312 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:32:26.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:32:26.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA6y+lmOBjXAxAAH99rCH3EVdvdjP0h57xfWA== --osd-uuid c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:32:26.366 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:26.368+0000 7f9e0f7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:26.388 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:26.388+0000 7f9e0f7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:26.389 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:26.388+0000 7f9e0f7dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:26.389 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:26.388+0000 7f9e0f7dc7c0 -1 bdev(0x55e5f568dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:32:26.390 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:26.388+0000 7f9e0f7dc7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:32:32.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:32:32.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:32:32.427 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:32:32.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:32:32.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:32:32.849 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:32:32.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:32:32.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:32:32.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:32:32.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:32:32.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:32:32.884 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:32.884+0000 7fdf608d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:32.903 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:32.904+0000 7fdf608d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:32.904 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:32.904+0000 7fdf608d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:32:33.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:33.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:33.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:34.441 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:34.440+0000 7fdf608d37c0 -1 Falling back to public interface 2024-09-17T18:32:34.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:34.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:34.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:32:34.543 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:32:34.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:34.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:34.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:35.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:35.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:35.859 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:32:35.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:32:35.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:35.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:36.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:37.065 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:32:37.064+0000 7fdf608d37c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:32:37.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:37.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:37.191 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:32:37.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:32:37.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:37.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:37.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:38.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:38.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:38.559 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:32:38.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:32:38.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:38.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:38.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:39.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:39.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:39.955 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:32:39.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:32:39.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:39.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:40.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:41.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:41.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:41.291 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:32:41.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:32:41.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:41.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:41.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:42.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:42.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:42.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:32:42.630 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:32:42.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:42.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:42.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:43.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:43.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:43.982 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:32:43.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:32:43.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:43.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:44.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:45.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:45.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:45.319 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:32:45.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:32:45.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:45.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:45.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:46.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:46.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:46.757 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:32:46.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:32:46.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:46.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:47.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:48.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:48.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:48.096 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:32:48.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:32:48.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:48.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:48.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:49.418 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:32:49.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:32:49.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:49.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:50.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:51.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:51.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:51.162 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:32:51.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:32:51.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:51.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:51.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:52.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:52.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:52.525 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:32:52.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:32:52.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:52.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:53.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:54.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:54.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:54.069 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:32:54.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:32:54.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:54.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:54.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:55.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:55.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:55.774 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:32:55.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:32:55.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:55.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:56.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:57.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:57.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:57.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:32:57.644 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:32:57.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:57.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:58.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:32:59.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:32:59.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:32:59.486 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:32:59.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:32:59.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:32:59.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:32:59.822 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3443048070,v1:127.0.0.1:6803/3443048070] [v2:127.0.0.1:6804/3443048070,v1:127.0.0.1:6805/3443048070] exists,up c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:32:59.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:32:59.825 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:32:59.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:32:59.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:32:59.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:32:59.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f1ac164e-241c-4704-b9f3-f337f099de1d 2024-09-17T18:32:59.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f1ac164e-241c-4704-b9f3-f337f099de1d' 2024-09-17T18:32:59.829 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 f1ac164e-241c-4704-b9f3-f337f099de1d 2024-09-17T18:32:59.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:32:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBby+lmKj+rMhAA1iFsHL0Cb+UukxAWD8DAzw== 2024-09-17T18:32:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBby+lmKj+rMhAA1iFsHL0Cb+UukxAWD8DAzw=="}' 2024-09-17T18:32:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f1ac164e-241c-4704-b9f3-f337f099de1d -i td/osd-scrub-repair/1/new.json 2024-09-17T18:33:00.338 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:33:00.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:33:00.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBby+lmKj+rMhAA1iFsHL0Cb+UukxAWD8DAzw== --osd-uuid f1ac164e-241c-4704-b9f3-f337f099de1d 2024-09-17T18:33:00.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:00.404+0000 7fb2fcb4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:00.473 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:00.472+0000 7fb2fcb4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:00.476 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:00.476+0000 7fb2fcb4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:00.476 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:00.476+0000 7fb2fcb4b7c0 -1 bdev(0x563f243fc800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:33:00.476 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:00.476+0000 7fb2fcb4b7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:33:06.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:33:06.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:33:06.588 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:33:06.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:33:06.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:33:06.993 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:33:06.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:33:06.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:33:06.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:33:06.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:33:06.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:33:07.028 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:07.027+0000 7fbd518ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:07.052 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:07.051+0000 7fbd518ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:07.053 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:07.051+0000 7fbd518ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:07.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:33:07.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:33:07.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:33:07.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:33:07.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:33:07.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:07.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:33:07.335 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:33:07.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:07.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:07.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:08.569 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:08.567+0000 7fbd518ac7c0 -1 Falling back to public interface 2024-09-17T18:33:08.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:08.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:08.636 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:33:08.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:33:08.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:08.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:08.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:09.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:09.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:09.972 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:33:09.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:33:09.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:09.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:10.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:11.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:11.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:11.307 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:33:11.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:33:11.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:11.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:11.334 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:11.335+0000 7fbd518ac7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:33:11.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:12.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:12.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:12.654 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:33:12.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:33:12.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:12.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:12.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:13.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:13.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:13.976 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:33:13.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:33:13.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:13.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:14.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:15.373 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:33:15.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:33:15.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:15.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:15.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:16.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:16.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:16.694 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:33:16.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:33:16.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:16.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:17.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:18.017 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:18.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:19.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:19.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:19.367 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:33:19.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:33:19.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:19.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:19.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:20.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:20.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:20.689 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:33:20.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:33:20.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:20.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:21.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:22.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:22.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:22.031 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:33:22.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:33:22.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:22.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:22.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:23.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:23.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:23.367 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:33:23.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:33:23.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:23.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:23.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:24.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:24.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:24.703 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:33:24.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:33:24.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:24.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:26.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:26.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:26.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:33:26.662 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:33:26.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:26.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:27.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:28.130 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:33:28.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:33:28.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:28.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:28.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:29.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:29.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:29.853 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:33:29.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:33:29.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:29.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:30.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:31.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:31.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:31.431 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:33:31.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:33:31.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:31.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:32.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:33.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:33.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:33.064 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:33:33.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:33:33.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:33.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:33.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:34.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:34.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:34.403 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:33:34.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:33:34.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:34.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:33:34.765 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2318146705,v1:127.0.0.1:6811/2318146705] [v2:127.0.0.1:6812/2318146705,v1:127.0.0.1:6813/2318146705] exists,up f1ac164e-241c-4704-b9f3-f337f099de1d 2024-09-17T18:33:34.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:33:34.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:33:34.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:33:34.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:33:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:33:34.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:33:34.772 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:33:34.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=26092322-2566-4058-958e-9fd65a05f1ad 2024-09-17T18:33:34.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 26092322-2566-4058-958e-9fd65a05f1ad' 2024-09-17T18:33:34.772 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 26092322-2566-4058-958e-9fd65a05f1ad 2024-09-17T18:33:34.772 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:33:34.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB+y+lmov6ALxAA8uN6i7Z7rHuQaIgHtfjTzA== 2024-09-17T18:33:34.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB+y+lmov6ALxAA8uN6i7Z7rHuQaIgHtfjTzA=="}' 2024-09-17T18:33:34.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 26092322-2566-4058-958e-9fd65a05f1ad -i td/osd-scrub-repair/2/new.json 2024-09-17T18:33:35.191 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:33:35.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:33:35.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB+y+lmov6ALxAA8uN6i7Z7rHuQaIgHtfjTzA== --osd-uuid 26092322-2566-4058-958e-9fd65a05f1ad 2024-09-17T18:33:35.249 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:35.247+0000 7f35b04347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:35.285 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:35.283+0000 7f35b04347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:35.287 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:35.287+0000 7f35b04347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:35.287 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:35.287+0000 7f35b04347c0 -1 bdev(0x559d6f8bbc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:33:35.288 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:35.287+0000 7f35b04347c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:33:41.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:33:41.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:33:41.132 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:33:41.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:33:41.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:33:41.547 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:33:41.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:33:41.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:33:41.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:33:41.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:33:41.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:33:41.583 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:41.583+0000 7f42261347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:41.614 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:41.615+0000 7f42261347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:41.616 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:41.615+0000 7f42261347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:33:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:33:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:33:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:33:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:33:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:33:41.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:41.893 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:33:41.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:33:41.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:41.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:42.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:43.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:43.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:43.232 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:33:43.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:33:43.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:43.408 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:43.407+0000 7f42261347c0 -1 Falling back to public interface 2024-09-17T18:33:43.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:44.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:44.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:44.575 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:33:44.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:33:44.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:44.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:44.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:45.904 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:33:45.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:33:45.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:45.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:46.070 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:33:46.071+0000 7f42261347c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:33:46.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:47.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:47.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:47.274 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:33:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:33:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:47.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:48.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:48.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:48.749 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:33:48.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:33:48.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:48.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:49.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:50.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:50.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:50.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:33:50.211 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:33:50.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:50.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:50.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:33:51.545 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:33:51.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:51.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:51.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:52.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:52.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:52.958 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:33:52.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:33:52.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:52.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:53.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:54.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:54.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:54.284 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:33:54.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:33:54.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:54.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:54.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:55.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:55.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:55.614 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:33:55.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:33:55.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:55.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:55.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:56.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:56.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:56.956 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:33:56.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:33:56.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:56.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:57.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:58.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:58.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:58.415 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:33:58.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:33:58.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:58.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:33:58.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:33:59.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:33:59.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:33:59.760 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:33:59.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:33:59.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:33:59.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:00.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:01.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:01.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:01.161 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:34:01.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:34:01.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:01.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:01.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:02.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:02.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:02.545 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:34:02.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:34:02.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:02.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:02.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:03.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:03.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:03.981 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:34:03.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:34:03.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:03.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:04.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:05.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:05.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:05.528 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:34:05.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:34:05.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:05.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:06.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:07.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:07.116 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:34:07.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:07.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:34:07.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:07.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:07.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:08.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:08.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:08.858 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:34:08.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:34:08.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:08.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:09.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:10.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:10.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:10.495 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:34:10.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:34:10.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:10.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:34:10.827 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/639653206,v1:127.0.0.1:6819/639653206] [v2:127.0.0.1:6820/639653206,v1:127.0.0.1:6821/639653206] exists,up 26092322-2566-4058-958e-9fd65a05f1ad 2024-09-17T18:34:10.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:34:10.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:34:10.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:34:10.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-09-17T18:34:10.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-17T18:34:11.251 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'testpool' created 2024-09-17T18:34:11.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:34:12.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-09-17T18:34:12.716 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 1 size to 2 2024-09-17T18:34:12.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-17T18:34:12.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:34:12.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:34:12.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:34:12.738 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:34:12.739 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:34:12.739 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:34:12.739 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:34:12.739 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:34:12.739 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:34:12.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:34:12.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:34:12.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:34:12.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:34:12.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:34:12.853 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:34:13.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:34:13.279 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:13.279 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:34:13.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:34:13.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:13.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:34:13.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836498 2024-09-17T18:34:13.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836498 2024-09-17T18:34:13.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498' 2024-09-17T18:34:13.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:13.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:34:13.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672972 2024-09-17T18:34:13.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672972 2024-09-17T18:34:13.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-09-17T18:34:13.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:13.571 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:34:13.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509445 2024-09-17T18:34:13.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509445 2024-09-17T18:34:13.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509445' 2024-09-17T18:34:13.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:13.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836498 2024-09-17T18:34:13.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:13.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:34:13.720 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836498 2024-09-17T18:34:13.720 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:13.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836498 2024-09-17T18:34:13.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-17T18:34:13.722 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836498 2024-09-17T18:34:13.722 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836498 2024-09-17T18:34:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:34:15.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:34:15.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:15.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-17T18:34:15.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:15.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672972 2024-09-17T18:34:15.423 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:15.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:34:15.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:15.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672972 2024-09-17T18:34:15.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672972 2024-09-17T18:34:15.427 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672972 2024-09-17T18:34:15.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-17T18:34:15.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:34:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-17T18:34:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:15.751 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509445 2024-09-17T18:34:15.751 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:15.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:34:15.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509445 2024-09-17T18:34:15.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:15.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509445 2024-09-17T18:34:15.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-17T18:34:15.754 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509445 2024-09-17T18:34:15.754 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:34:16.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509445 2024-09-17T18:34:16.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:34:16.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:16.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:16.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:34:16.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:34:16.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:34:16.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:34:16.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:34:16.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:34:16.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:34:16.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:34:16.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:34:16.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:34:16.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:16.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:34:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:34:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:34:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-09-17T18:34:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-09-17T18:34:17.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-09-17T18:34:17.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:17.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:17.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:17.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:17.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:17.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:17.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:17.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:17.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:17.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:17.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:17.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:18.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:18.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:18.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:18.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:18.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:18.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:18.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-09-17T18:34:18.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:34:18.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-17T18:34:18.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:34:18.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:34:18.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:34:18.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:34:18.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:34:18.683 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:34:18.683 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:34:19.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:34:19.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:34:19.025 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:34:19.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:34:19.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:34:19.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:34:19.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:34:19.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:34:19.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:34:19.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-17T18:34:19.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:34:19.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:34:19.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:34:19.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:34:19.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:34:19.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-17T18:34:21.374 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-17T18:34:21.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:34:21.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:34:21.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:34:21.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:34:21.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:34:21.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:34:21.912 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:34:21.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:34:21.913 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:34:21.913 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:34:21.913 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:34:21.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:34:21.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:34:21.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:34:21.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:34:21.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:34:21.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:34:21.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:34:21.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:34:21.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:34:21.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:34:21.956 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:21.954+0000 7f5ee8d9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:21.959 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:21.958+0000 7f5ee8d9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:21.960 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:21.958+0000 7f5ee8d9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:34:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:34:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:34:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:34:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:34:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:22.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:34:22.247 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:34:22.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:22.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:22.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:23.014+0000 7f5ee8d9c7c0 -1 Falling back to public interface 2024-09-17T18:34:23.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:23.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:23.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:34:23.594 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:23.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:23.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:23.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:24.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:24.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:24.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:34:24.914 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:34:24.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:24.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:25.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:25.770 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:25.770+0000 7f5ee8d9c7c0 -1 osd.0 20 log_to_monitors true 2024-09-17T18:34:26.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:26.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:26.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:34:26.239 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:34:26.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:26.240 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1160503030,v1:127.0.0.1:6803/1160503030] [v2:127.0.0.1:6804/1160503030,v1:127.0.0.1:6805/1160503030] exists,up c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:34:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:34:26.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:34:26.675 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:34:26.675 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:34:26.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:34:26.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:34:26.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:34:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:34:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:34:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:34:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:34:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:34:26.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:34:27.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:34:27.156 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:27.156 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:34:27.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:34:27.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:27.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:34:27.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=103079215106 2024-09-17T18:34:27.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 103079215106 2024-09-17T18:34:27.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106' 2024-09-17T18:34:27.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:27.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:34:27.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-17T18:34:27.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-17T18:34:27.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672976' 2024-09-17T18:34:27.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:27.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:34:27.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-17T18:34:27.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-17T18:34:27.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-103079215106 1-42949672976 2-64424509449' 2024-09-17T18:34:27.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:27.581 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-103079215106 2024-09-17T18:34:27.581 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:34:27.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-103079215106 2024-09-17T18:34:27.583 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:27.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=103079215106 2024-09-17T18:34:27.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-09-17T18:34:27.584 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 103079215106 2024-09-17T18:34:27.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:27.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 103079215106 2024-09-17T18:34:27.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:34:28.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:34:28.893 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:29.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 103079215106 -lt 103079215106 2024-09-17T18:34:29.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:29.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-17T18:34:29.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:29.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:34:29.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-17T18:34:29.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:29.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-17T18:34:29.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-17T18:34:29.226 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672976 2024-09-17T18:34:29.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:34:29.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-17T18:34:29.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:29.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-17T18:34:29.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:29.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:34:29.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-17T18:34:29.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:29.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-17T18:34:29.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-17T18:34:29.573 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509449 2024-09-17T18:34:29.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:34:29.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-17T18:34:29.919 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:34:29.919 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:29.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:30.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:34:30.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:34:30.301 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:34:30.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:34:30.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:34:30.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:34:30.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:34:30.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:34:30.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:34:30.651 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:34:30.651 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:30.651 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:31.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:34:31.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:34:31.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:34:31.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-09-17T18:34:31.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:34:31.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:34:31.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:34:31.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:34:31.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:34:31.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:34:31.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:34:31.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:34:31.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:34:31.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:34:31.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-17T18:34:31.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:34:31.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:34:31.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:34:31.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:34:31.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-17T18:34:31.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:34:31.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:34:31.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:34:31.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:34:31.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:34:31.707 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:34:31.707 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:34:31.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:34:31.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:34:31.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:34:32.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-17T18:34:33.531 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:104778fc:::obj2:head# 2024-09-17T18:34:34.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:34:34.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:34:34.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:34:34.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:34:34.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:34:34.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:34:34.075 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:34:34.076 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:34:34.076 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:34:34.077 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:34:34.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:34:34.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:34:34.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:34:34.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:34:34.078 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:34:34.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:34:34.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:34:34.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:34:34.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:34:34.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:34:34.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:34:34.114 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:34.114+0000 7fb208d707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:34.117 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:34.114+0000 7fb208d707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:34.118 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:34.118+0000 7fb208d707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:34.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:34:34.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:34.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:34.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:35.413 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:35.410+0000 7fb208d707c0 -1 Falling back to public interface 2024-09-17T18:34:35.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:35.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:35.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:34:35.750 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:35.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:35.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:36.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:37.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:37.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:37.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:34:37.071 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:34:37.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:37.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:37.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:38.059 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:38.058+0000 7fb208d707c0 -1 osd.0 25 log_to_monitors true 2024-09-17T18:34:38.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:38.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:38.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:34:38.432 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:34:38.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:38.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:34:38.821 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/843439767,v1:127.0.0.1:6803/843439767] [v2:127.0.0.1:6804/843439767,v1:127.0.0.1:6805/843439767] exists,up c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:34:38.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:34:38.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:34:38.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:34:38.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:34:38.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:34:38.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:34:38.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:34:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:34:38.823 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:34:38.823 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:34:38.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:34:38.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:34:38.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:34:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:34:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:34:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:34:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:34:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:34:38.932 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:34:39.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:34:39.341 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:39.341 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:34:39.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:34:39.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:39.342 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:34:39.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051586 2024-09-17T18:34:39.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051586 2024-09-17T18:34:39.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586' 2024-09-17T18:34:39.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:39.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:34:39.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-17T18:34:39.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-17T18:34:39.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-42949672979' 2024-09-17T18:34:39.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:39.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:34:39.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-17T18:34:39.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-17T18:34:39.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051586 1-42949672979 2-64424509452' 2024-09-17T18:34:39.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:39.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051586 2024-09-17T18:34:39.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:39.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:34:39.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:39.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051586 2024-09-17T18:34:39.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051586 2024-09-17T18:34:39.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-09-17T18:34:39.868 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 124554051586 2024-09-17T18:34:39.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:40.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051586 2024-09-17T18:34:40.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:34:41.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:34:41.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:41.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051586 -lt 124554051586 2024-09-17T18:34:41.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:41.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-17T18:34:41.524 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:41.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:34:41.526 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-17T18:34:41.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:41.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-17T18:34:41.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-17T18:34:41.529 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672979 2024-09-17T18:34:41.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:34:41.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672979 2024-09-17T18:34:41.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:41.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-17T18:34:41.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:41.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:34:41.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:41.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-17T18:34:41.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-17T18:34:41.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-17T18:34:41.877 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509452 2024-09-17T18:34:41.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:34:42.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-17T18:34:42.212 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:34:42.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:42.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:42.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:34:42.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:34:42.632 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:34:42.632 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:34:42.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:34:42.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:34:42.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:34:42.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:34:43.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:34:43.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:34:43.003 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:43.003 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:43.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:34:43.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:34:43.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:34:43.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-17T18:34:43.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:34:43.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:34:43.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:34:43.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:34:43.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-17T18:34:43.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:34:43.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:34:43.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:34:43.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:34:43.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:34:43.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:34:43.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:34:44.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-17T18:34:46.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:34:46.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:34:46.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:34:46.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:34:46.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:34:46.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:34:46.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:34:46.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:34:46.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:34:46.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:34:46.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:34:46.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:34:46.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:34:46.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:34:46.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:34:46.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:34:46.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:34:46.256 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:34:46.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:34:46.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:34:46.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:34:46.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:34:46.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:34:46.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:34:46.299 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:46.302+0000 7f181c5f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:46.302 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:46.306+0000 7f181c5f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:46.304 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:46.306+0000 7f181c5f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:34:46.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:47.565 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:47.566+0000 7f181c5f87c0 -1 Falling back to public interface 2024-09-17T18:34:47.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:47.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:47.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:34:47.939 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:47.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:47.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:34:48.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:49.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:49.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:49.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:34:49.268 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:34:49.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:49.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:34:49.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:34:50.200 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:34:50.201+0000 7f181c5f87c0 -1 osd.1 30 log_to_monitors true 2024-09-17T18:34:50.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:34:50.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:34:50.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:34:50.592 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:34:50.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:34:50.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:34:50.981 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/3685806230,v1:127.0.0.1:6811/3685806230] [v2:127.0.0.1:6812/3685806230,v1:127.0.0.1:6813/3685806230] exists,up f1ac164e-241c-4704-b9f3-f337f099de1d 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:34:50.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:34:50.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:34:50.983 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:34:50.983 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:34:50.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:34:50.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:34:50.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:34:51.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:34:51.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:34:51.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:34:51.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:34:51.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:34:51.076 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:34:51.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:34:51.399 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:34:51.399 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:34:51.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:34:51.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:51.399 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:34:51.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051589 2024-09-17T18:34:51.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051589 2024-09-17T18:34:51.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051589' 2024-09-17T18:34:51.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:51.548 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:34:51.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888066 2024-09-17T18:34:51.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888066 2024-09-17T18:34:51.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051589 1-146028888066' 2024-09-17T18:34:51.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:34:51.691 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:34:51.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-17T18:34:51.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-17T18:34:51.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051589 1-146028888066 2-64424509456' 2024-09-17T18:34:51.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:51.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051589 2024-09-17T18:34:51.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:51.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:34:51.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051589 2024-09-17T18:34:51.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:51.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051589 2024-09-17T18:34:51.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-09-17T18:34:51.829 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 124554051589 2024-09-17T18:34:51.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:52.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051588 -lt 124554051589 2024-09-17T18:34:52.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:34:53.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:34:53.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:34:53.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051589 -lt 124554051589 2024-09-17T18:34:53.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:53.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-146028888066 2024-09-17T18:34:53.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:53.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:34:53.475 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-146028888066 2024-09-17T18:34:53.475 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:53.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888066 2024-09-17T18:34:53.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2024-09-17T18:34:53.476 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 146028888066 2024-09-17T18:34:53.476 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:34:53.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888066 -lt 146028888066 2024-09-17T18:34:53.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:34:53.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-17T18:34:53.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:34:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:34:53.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-17T18:34:53.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:34:53.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-17T18:34:53.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-17T18:34:53.820 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509456 2024-09-17T18:34:53.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:34:54.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509456 2024-09-17T18:34:54.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:34:54.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:54.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:54.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:34:54.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:34:54.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:34:54.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:34:54.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:34:54.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:34:54.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:34:54.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:34:54.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:34:54.904 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:34:54.905 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:34:54.905 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:34:55.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:34:55.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:34:55.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:34:55.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-09-17T18:34:55.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-17T18:34:55.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-17T18:34:55.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-17T18:34:55.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:34:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-09-17T18:34:55.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-09-17T18:34:55.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:34:55.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-17T18:34:55.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-17T18:34:55.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:34:55.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-09-17T18:34:55.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-09-17T18:34:55.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:34:55.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:34:55.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:34:55.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:34:56.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-09-17T18:34:11.206673+0000 2024-09-17T18:34:56.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-09-17T18:34:56.457 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:34:56.457 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": true, 2024-09-17T18:34:56.457 INFO:tasks.workunit.client.0.smithi105.stdout: "must": false, 2024-09-17T18:34:56.457 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "2024-09-03T18:33:16.460719+0000" 2024-09-17T18:34:56.457 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:34:56.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-09-17T18:34:11.206673+0000 2024-09-17T18:34:56.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-17T18:34:56.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:34:11.206673+0000 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:34:56.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:34:56.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:11.206673+0000 '>' 2024-09-17T18:34:11.206673+0000 2024-09-17T18:34:56.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:34:57.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:34:57.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:34:57.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:34:57.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:34:57.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:34:57.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:34:57.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:34:58.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:11.206673+0000 '>' 2024-09-17T18:34:11.206673+0000 2024-09-17T18:34:58.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:34:59.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:34:59.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:34:59.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:34:59.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:34:59.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:34:59.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:34:59.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:34:59.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:11.206673+0000 '>' 2024-09-17T18:34:11.206673+0000 2024-09-17T18:34:59.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:35:00.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:35:00.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:35:00.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:35:00.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:00.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:00.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:00.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:59.387850+0000 '>' 2024-09-17T18:34:11.206673+0000 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:35:00.768 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:35:00.769 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:35:00.769 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:35:00.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:35:00.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:35:00.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:35:00.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:35:00.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:35:00.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:35:00.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:35:00.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:35:00.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:35:01.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:35:01.222 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:01.222 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:35:01.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:35:01.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:01.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:35:01.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051592 2024-09-17T18:35:01.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051592 2024-09-17T18:35:01.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051592' 2024-09-17T18:35:01.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:01.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:35:01.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888069 2024-09-17T18:35:01.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888069 2024-09-17T18:35:01.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051592 1-146028888069' 2024-09-17T18:35:01.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:01.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:35:01.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-17T18:35:01.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-17T18:35:01.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051592 1-146028888069 2-64424509459' 2024-09-17T18:35:01.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:01.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:01.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051592 2024-09-17T18:35:01.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:35:01.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051592 2024-09-17T18:35:01.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:01.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051592 2024-09-17T18:35:01.676 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 124554051592 2024-09-17T18:35:01.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051592' 2024-09-17T18:35:01.676 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:01.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051591 -lt 124554051592 2024-09-17T18:35:01.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:35:02.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:35:02.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:03.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051592 -lt 124554051592 2024-09-17T18:35:03.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:03.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-146028888069 2024-09-17T18:35:03.310 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:03.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:35:03.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-146028888069 2024-09-17T18:35:03.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:03.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888069 2024-09-17T18:35:03.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2024-09-17T18:35:03.313 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 146028888069 2024-09-17T18:35:03.313 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:35:03.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888069 -lt 146028888069 2024-09-17T18:35:03.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:03.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-17T18:35:03.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:03.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:35:03.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-17T18:35:03.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:03.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-17T18:35:03.635 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509459 2024-09-17T18:35:03.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-17T18:35:03.635 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:35:03.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-17T18:35:03.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:35:03.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:35:03.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:35:04.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:35:04.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:35:04.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:35:04.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:35:04.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:35:04.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:35:04.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:35:04.726 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:35:04.726 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:35:04.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:35:05.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:35:05.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:35:05.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:35:05.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-17T18:35:05.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:35:05.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:35:05.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:35:05.444 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:05.444 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:35:05.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:35:05.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:05.444 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:35:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051594 2024-09-17T18:35:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051594 2024-09-17T18:35:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051594' 2024-09-17T18:35:05.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:05.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:35:05.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888071 2024-09-17T18:35:05.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888071 2024-09-17T18:35:05.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051594 1-146028888071' 2024-09-17T18:35:05.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:05.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:35:05.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-17T18:35:05.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-17T18:35:05.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051594 1-146028888071 2-64424509461' 2024-09-17T18:35:05.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:05.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051594 2024-09-17T18:35:05.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:05.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:35:05.893 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051594 2024-09-17T18:35:05.893 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:05.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051594 2024-09-17T18:35:05.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-09-17T18:35:05.894 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 124554051594 2024-09-17T18:35:05.894 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:06.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051593 -lt 124554051594 2024-09-17T18:35:06.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:35:07.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:35:07.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:07.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051594 -lt 124554051594 2024-09-17T18:35:07.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:07.524 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-146028888071 2024-09-17T18:35:07.524 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:07.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:35:07.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-146028888071 2024-09-17T18:35:07.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:07.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888071 2024-09-17T18:35:07.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 146028888071' 2024-09-17T18:35:07.528 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 146028888071 2024-09-17T18:35:07.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:35:07.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888071 -lt 146028888071 2024-09-17T18:35:07.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:07.852 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-17T18:35:07.852 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:07.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:35:07.854 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-17T18:35:07.855 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:07.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-17T18:35:07.856 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509461 2024-09-17T18:35:07.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-17T18:35:07.856 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:35:08.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-17T18:35:08.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-17T18:35:08.190 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:08.191 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-17T18:35:08.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-09-17T18:35:08.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-17T18:35:08.497 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:35:08.497 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-17T18:34:59.387984+0000 20'10 35:110 [1,0] 1 [1,0] 1 20'10 2024-09-17T18:34:59.387850+0000 20'10 2024-09-17T18:34:59.387850+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:34:59.387850+0000 10 0 2024-09-17T18:35:08.498 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:35:08.498 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:35:08.498 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:35:08.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-17T18:35:08.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-09-17T18:35:08.825 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:35:08.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-09-17T18:35:08.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:35:08.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-17T18:35:08.843 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2024-09-17T18:35:08.844 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:35:08.844 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-17T18:35:08.844 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:35:08.844 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-17T18:35:09.158 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:35:09.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2024-09-17T18:35:09.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:35:09.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-09-17T18:35:09.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:35:09.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:35:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:35:09.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:35:09.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-09-17T18:35:09.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:35:09.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:35:09.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:35:09.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:35:09.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:35:09.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-09-17T18:35:10.342 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:35:10.343 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:35:10.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:35:10.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:35:10.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:10.644 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:35:10.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:35:10.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:35:10.647 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:35:10.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:35:10.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:35:10.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:35:10.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:35:10.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:35:10.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:35:10.682 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:10.685+0000 7fab8c50d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:10.685 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:10.685+0000 7fab8c50d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:10.686 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:10.689+0000 7fab8c50d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:10.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:11.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:12.009 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:12.009+0000 7fab8c50d7c0 -1 Falling back to public interface 2024-09-17T18:35:12.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:12.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:12.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:35:12.347 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:12.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:12.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:12.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:13.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:13.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:13.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:35:13.674 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:35:13.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:13.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:13.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:14.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:14.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:14.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:35:14.994 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:35:14.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:14.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:15.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:15.399 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:15.401+0000 7fab8c50d7c0 -1 osd.0 35 log_to_monitors true 2024-09-17T18:35:16.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:16.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:16.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:35:16.331 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:35:16.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:16.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 39 up_thru 31 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6802/4035338813,v1:127.0.0.1:6803/4035338813] [v2:127.0.0.1:6804/4035338813,v1:127.0.0.1:6805/4035338813] exists,up c44bc72c-483a-4594-9a03-3f9f0e69fb58 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:35:16.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:35:16.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:35:16.665 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:35:16.666 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:35:16.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:35:16.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:35:16.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:35:16.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:35:16.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:35:16.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:35:16.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:35:16.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:35:16.776 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:35:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:35:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:35:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:35:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:17.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:35:17.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724546 2024-09-17T18:35:17.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724546 2024-09-17T18:35:17.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724546' 2024-09-17T18:35:17.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:17.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:35:17.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=146028888074 2024-09-17T18:35:17.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 146028888074 2024-09-17T18:35:17.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724546 1-146028888074' 2024-09-17T18:35:17.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:17.478 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:35:17.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-17T18:35:17.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-17T18:35:17.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724546 1-146028888074 2-64424509464' 2024-09-17T18:35:17.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:17.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-167503724546 2024-09-17T18:35:17.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:17.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:35:17.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-167503724546 2024-09-17T18:35:17.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:17.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724546 2024-09-17T18:35:17.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 167503724546' 2024-09-17T18:35:17.624 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 167503724546 2024-09-17T18:35:17.625 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:17.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 167503724546 2024-09-17T18:35:17.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:35:18.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:35:18.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:19.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724546 -lt 167503724546 2024-09-17T18:35:19.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:19.270 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-146028888074 2024-09-17T18:35:19.270 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:19.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:35:19.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-146028888074 2024-09-17T18:35:19.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:19.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=146028888074 2024-09-17T18:35:19.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 146028888074' 2024-09-17T18:35:19.274 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 146028888074 2024-09-17T18:35:19.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:35:19.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 146028888074 -lt 146028888074 2024-09-17T18:35:19.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:19.627 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-17T18:35:19.627 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:19.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:35:19.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-17T18:35:19.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:19.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-17T18:35:19.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-17T18:35:19.633 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509464 2024-09-17T18:35:19.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:35:19.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-17T18:35:19.946 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:35:19.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:35:19.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:35:20.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:35:20.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:35:20.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:35:20.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:35:20.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:35:20.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:35:20.359 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:35:20.359 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:35:20.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:35:20.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:35:20.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:35:20.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:35:21.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:35:21.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:35:21.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:35:21.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-09-17T18:35:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:35:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-17T18:35:21.093 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:21.094 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-17T18:35:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-09-17T18:35:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:35:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:35:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:35:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:35:21.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:35:21.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:35:21.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:35:21.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:35:21.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:35:21.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:35:21.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:35:21.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-09-17T18:35:23.586 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-09-17T18:35:23.586 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:104778fc:::obj2:head# 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:35:24.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:24.123 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:24.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:24.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:24.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:35:24.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:35:24.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:35:24.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:35:24.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:35:24.126 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:35:24.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:35:24.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:35:24.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:35:24.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:35:24.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:35:24.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:35:24.162 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:24.165+0000 7faaf53de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:24.165 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:24.165+0000 7faaf53de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:24.166 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:24.169+0000 7faaf53de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:24.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:24.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:35:24.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:25.444 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:25.445+0000 7faaf53de7c0 -1 Falling back to public interface 2024-09-17T18:35:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:35:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:35:26.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:27.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:27.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:27.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:35:27.122 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:35:27.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:27.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:35:27.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:28.326 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:28.329+0000 7faaf53de7c0 -1 osd.1 40 log_to_monitors true 2024-09-17T18:35:28.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:28.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:28.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:35:28.461 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:35:28.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:28.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:35:28.801 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 44 up_thru 39 down_at 41 last_clean_interval [34,40) [v2:127.0.0.1:6810/2705689507,v1:127.0.0.1:6811/2705689507] [v2:127.0.0.1:6812/2705689507,v1:127.0.0.1:6813/2705689507] exists,up f1ac164e-241c-4704-b9f3-f337f099de1d 2024-09-17T18:35:28.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:35:28.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:35:28.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:35:28.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:35:28.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:35:28.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:35:28.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:35:28.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:35:28.803 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:35:28.803 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:35:28.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:35:28.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:35:28.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:35:28.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:35:28.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:35:28.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:35:28.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:35:28.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:35:28.912 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:35:29.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:35:29.246 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:29.246 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:35:29.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:35:29.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:29.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:35:29.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724549 2024-09-17T18:35:29.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724549 2024-09-17T18:35:29.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724549' 2024-09-17T18:35:29.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:29.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:35:29.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561026 2024-09-17T18:35:29.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561026 2024-09-17T18:35:29.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724549 1-188978561026' 2024-09-17T18:35:29.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:29.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:35:29.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-17T18:35:29.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-17T18:35:29.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724549 1-188978561026 2-64424509467' 2024-09-17T18:35:29.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:29.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-167503724549 2024-09-17T18:35:29.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:29.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:35:29.745 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-167503724549 2024-09-17T18:35:29.745 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:29.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724549 2024-09-17T18:35:29.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 167503724549' 2024-09-17T18:35:29.746 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 167503724549 2024-09-17T18:35:29.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:30.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724548 -lt 167503724549 2024-09-17T18:35:30.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:35:31.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:35:31.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:31.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724549 -lt 167503724549 2024-09-17T18:35:31.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:31.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-188978561026 2024-09-17T18:35:31.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:31.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:35:31.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-188978561026 2024-09-17T18:35:31.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:31.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561026 2024-09-17T18:35:31.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 188978561026' 2024-09-17T18:35:31.442 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 188978561026 2024-09-17T18:35:31.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:35:31.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561026 -lt 188978561026 2024-09-17T18:35:31.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:31.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-17T18:35:31.799 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:31.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:35:31.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-17T18:35:31.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:31.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-17T18:35:31.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-17T18:35:31.803 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509467 2024-09-17T18:35:31.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:35:32.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509467 2024-09-17T18:35:32.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:35:32.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:35:32.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:35:32.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:35:32.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:35:32.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:35:32.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:35:32.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:35:32.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:35:32.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:35:32.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:35:32.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:35:32.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:35:32.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:35:32.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:35:33.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:35:33.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:35:33.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:35:33.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-09-17T18:35:33.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.0 2024-09-17T18:35:33.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.0 2024-09-17T18:35:33.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:33.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:33.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:33.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:33.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:33.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.0 2024-09-17T18:35:33.856 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 1.0 on osd.1 to repair 2024-09-17T18:35:33.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.0 2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:33.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-17T18:35:33.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:33.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:35:33.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:35:33.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:35:33.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:35:33.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:33.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:33.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:33.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:34.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:59.387850+0000 '>' 2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:34.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:35:35.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:35:35.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:35:35.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:35:35.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:35.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:35.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:35.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:35.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:59.387850+0000 '>' 2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:35.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:35:36.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:35:36.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:35:36.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:35:36.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:36.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:36.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:36.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:36.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:59.387850+0000 '>' 2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:36.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:35:37.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:35:37.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:35:37.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:35:37.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:37.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:37.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:37.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:38.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:34:59.387850+0000 '>' 2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:38.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:35:39.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:35:39.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:35:39.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:35:39.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:35:39.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:35:39.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:35:39.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:35:39.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:35:34.453293+0000 '>' 2024-09-17T18:34:59.387850+0000 2024-09-17T18:35:39.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:35:39.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:35:41.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-09-17T18:35:43.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-09-17T18:35:43.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:35:43.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:35:43.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:35:43.789 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:35:43.789 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:35:43.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:35:43.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:43.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:35:43.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724553 2024-09-17T18:35:43.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724553 2024-09-17T18:35:43.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724553' 2024-09-17T18:35:43.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:43.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:35:44.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=188978561030 2024-09-17T18:35:44.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 188978561030 2024-09-17T18:35:44.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724553 1-188978561030' 2024-09-17T18:35:44.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:35:44.090 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:35:44.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509471 2024-09-17T18:35:44.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509471 2024-09-17T18:35:44.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-167503724553 1-188978561030 2-64424509471' 2024-09-17T18:35:44.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:44.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-167503724553 2024-09-17T18:35:44.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:44.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:35:44.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-167503724553 2024-09-17T18:35:44.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:44.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724553 2024-09-17T18:35:44.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 167503724553' 2024-09-17T18:35:44.235 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 167503724553 2024-09-17T18:35:44.236 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:44.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724552 -lt 167503724553 2024-09-17T18:35:44.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:35:45.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:35:45.542 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:35:45.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724553 -lt 167503724553 2024-09-17T18:35:45.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:45.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-188978561030 2024-09-17T18:35:45.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:45.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:35:45.881 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-188978561030 2024-09-17T18:35:45.881 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:45.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=188978561030 2024-09-17T18:35:45.883 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 188978561030 2024-09-17T18:35:45.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 188978561030' 2024-09-17T18:35:45.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:35:46.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 188978561030 -lt 188978561030 2024-09-17T18:35:46.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:35:46.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509471 2024-09-17T18:35:46.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:35:46.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:35:46.216 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509471 2024-09-17T18:35:46.216 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:35:46.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509471 2024-09-17T18:35:46.217 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509471 2024-09-17T18:35:46.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-09-17T18:35:46.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:35:46.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509471 -lt 64424509471 2024-09-17T18:35:46.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-17T18:35:46.842 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:35:46.842 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-09-17T18:35:34.453427+0000 20'10 45:150 [1,0] 1 [1,0] 1 20'10 2024-09-17T18:35:34.453293+0000 20'10 2024-09-17T18:35:34.453293+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:35:34.453293+0000 9 0 2024-09-17T18:35:46.842 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:35:46.842 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:35:46.843 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:35:46.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-09-17T18:35:46.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-09-17T18:35:47.153 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:35:47.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-17T18:35:47.172 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:35:47.173 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:47.173 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-09-17T18:35:47.173 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 20 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-09-17T18:35:47.173 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:58.337+0000 7f1800582640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:47.173 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 15 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish: 3 errors. 0 errors fixed 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish 3 error(s) present with no repair possible 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 19 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR CHECK_REPAIR ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:34:59.385+0000 7f1800582640 10 osd.1 pg_epoch: 35 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=34/35 n=10 ec=16/16 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:35:34.449+0000 7faad9368640 10 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=10 ec=16/16 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:35:34.449+0000 7faad9368640 10 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=10 ec=16/16 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:35:34.449+0000 7faad9368640 15 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=16/16 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-17T18:35:47.174 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:35:34.449+0000 7faad9368640 20 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=16/16 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-09-17T18:35:47.175 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:35:34.449+0000 7faad9368640 19 osd.1 pg_epoch: 45 pg[1.0( v 20'10 (0'0,20'10] local-lis/les=44/45 n=9 ec=16/16 lis/c=44/44 les/c/f=45/45/0 sis=44) [1,0] r=0 lpr=44 crt=20'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:35:47.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:35:47.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:35:47.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:35:47.487 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:35:47.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:35:47.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:35:47.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:35:47.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:35:47.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:35:47.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:35:47.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:35:47.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:35:47.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:35:47.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:35:47.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:35:47.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:35:47.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:35:47.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:47.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:47.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:35:47.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:35:47.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:35:47.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:35:47.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:35:47.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:35:47.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:35:47.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:35:47.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:35:47.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:35:47.524 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:35:47.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:35:47.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:35:47.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:35:47.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:35:47.526 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:35:47.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:35:47.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:35:47.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:35:47.528 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:35:47.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:35:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:35:47.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:35:47.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:35:47.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:47.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:47.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:35:47.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:35:47.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:35:47.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:35:47.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:35:47.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:47.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:47.535 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:35:47.536 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:35:47.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:35:47.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:35:48.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:35:48.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:35:48.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:35:48.015 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:35:48.015 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:48.015 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:48.015 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:48.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:35:48.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:35:48.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:35:48.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:35:48.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:35:48.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:35:48.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:35:48.261 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:35:48.261 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:35:48.261 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:35:48.262 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:35:48.262 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:48.262 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:48.263 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:35:48.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:35:48.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:35:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:35:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:35:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:35:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:35:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:35:48.376 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:35:48.376 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:35:48.376 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:35:48.376 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:35:48.377 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:48.377 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:48.377 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:35:48.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:35:48.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:35:48.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-09-17T18:35:48.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:35:48.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:35:48.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:35:48.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:35:48.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:35:48.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:35:49.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:35:49.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:35:49.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:35:49.361 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:35:49.361 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:49.361 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:49.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:49.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:35:49.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:35:49.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:35:49.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-09-17T18:35:49.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-17T18:35:49.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:35:49.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:35:49.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:35:49.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:35:49.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:35:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:35:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:35:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:35:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:35:49.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:35:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=924eeb24-8b86-46db-9042-24a865d08e6b 2024-09-17T18:35:49.405 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 924eeb24-8b86-46db-9042-24a865d08e6b 2024-09-17T18:35:49.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 924eeb24-8b86-46db-9042-24a865d08e6b' 2024-09-17T18:35:49.405 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:35:49.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFzOlmHZNOGRAAnFSImcEYW3QEX+CDnH95MQ== 2024-09-17T18:35:49.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFzOlmHZNOGRAAnFSImcEYW3QEX+CDnH95MQ=="}' 2024-09-17T18:35:49.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 924eeb24-8b86-46db-9042-24a865d08e6b -i td/osd-scrub-repair/0/new.json 2024-09-17T18:35:49.695 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:35:49.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:35:49.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAFzOlmHZNOGRAAnFSImcEYW3QEX+CDnH95MQ== --osd-uuid 924eeb24-8b86-46db-9042-24a865d08e6b 2024-09-17T18:35:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:49.749+0000 7f62202807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:49.762 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:49.765+0000 7f62202807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:49.764 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:49.765+0000 7f62202807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:49.764 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:49.765+0000 7f62202807c0 -1 bdev(0x55d69dc01c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:35:49.765 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:49.765+0000 7f62202807c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:35:56.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:35:56.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:35:56.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:35:56.574 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:35:56.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:35:56.985 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:35:56.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:35:56.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:35:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:35:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:35:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:35:57.021 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:57.020+0000 7fd8550a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:57.060 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:57.060+0000 7fd8550a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:57.062 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:57.064+0000 7fd8550a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:35:57.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:35:57.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:35:57.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:35:57.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:35:57.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:35:57.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:57.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:35:57.282 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:35:57.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:57.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:57.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:58.076 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:35:58.076+0000 7fd8550a57c0 -1 Falling back to public interface 2024-09-17T18:35:58.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:58.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:58.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:35:58.584 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:35:58.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:58.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:35:58.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:35:59.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:35:59.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:35:59.900 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:35:59.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:35:59.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:35:59.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:00.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:00.966 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:00.968+0000 7fd8550a57c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:36:01.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:01.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:01.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:36:01.210 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:36:01.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:01.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:01.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:02.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:02.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:02.667 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:36:02.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:36:02.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:02.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:03.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:04.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:04.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:04.054 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:36:04.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:36:04.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:04.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:04.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:05.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:05.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:05.376 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:36:05.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:36:05.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:05.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:05.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:06.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:06.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:06.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:36:06.806 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:36:06.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:06.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:07.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:08.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:08.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:08.141 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:36:08.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:36:08.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:08.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:08.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:09.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:09.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:09.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:36:09.477 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:36:09.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:09.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:09.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:10.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:10.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:10.815 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:36:10.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:36:10.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:10.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:11.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:12.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:12.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:12.150 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:36:12.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:36:12.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:12.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:12.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:13.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:13.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:13.486 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:36:13.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:36:13.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:13.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:14.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:15.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:15.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:15.363 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:36:15.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:36:15.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:15.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:15.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:16.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:16.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:16.999 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:36:17.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:36:17.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:17.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:17.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:18.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:18.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:18.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:36:18.745 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:36:18.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:18.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:19.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:20.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:20.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:20.687 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:36:20.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:36:20.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:20.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:22.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:22.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:22.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:36:22.125 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:36:22.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:22.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:22.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:23.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:23.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:23.573 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:36:23.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:36:23.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:23.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:24.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:25.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:25.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:25.234 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:36:25.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:36:25.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:25.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:36:25.657 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2611468077,v1:127.0.0.1:6803/2611468077] [v2:127.0.0.1:6804/2611468077,v1:127.0.0.1:6805/2611468077] exists,up 924eeb24-8b86-46db-9042-24a865d08e6b 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:36:25.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:36:25.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:36:25.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:36:25.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:36:25.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:36:25.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:36:25.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:36:25.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:36:25.663 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 dc87d921-2456-48ff-ab7b-6ba3909c3779 2024-09-17T18:36:25.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc87d921-2456-48ff-ab7b-6ba3909c3779 2024-09-17T18:36:25.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 dc87d921-2456-48ff-ab7b-6ba3909c3779' 2024-09-17T18:36:25.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:36:25.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQApzOlmdLv8KBAAcY5LxtisF5Eri4BHwiaUkA== 2024-09-17T18:36:25.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQApzOlmdLv8KBAAcY5LxtisF5Eri4BHwiaUkA=="}' 2024-09-17T18:36:25.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc87d921-2456-48ff-ab7b-6ba3909c3779 -i td/osd-scrub-repair/1/new.json 2024-09-17T18:36:26.082 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:36:26.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:36:26.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQApzOlmdLv8KBAAcY5LxtisF5Eri4BHwiaUkA== --osd-uuid dc87d921-2456-48ff-ab7b-6ba3909c3779 2024-09-17T18:36:26.133 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:26.132+0000 7fc5f703a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:36:26.184 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:26.184+0000 7fc5f703a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:36:26.185 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:26.188+0000 7fc5f703a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:36:26.186 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:26.188+0000 7fc5f703a7c0 -1 bdev(0x564575a83c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:36:26.186 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:26.188+0000 7fc5f703a7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:36:32.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:36:32.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:36:32.237 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:36:32.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:36:32.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:36:32.644 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:36:32.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:36:32.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:36:32.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:36:32.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:36:32.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:36:32.678 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:32.680+0000 7f7b79e7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:36:32.712 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:32.712+0000 7f7b79e7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:36:32.714 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:32.716+0000 7f7b79e7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:36:33.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:36:33.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:36:33.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:36:33.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:36:33.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:36:33.008 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:36:33.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:33.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:36:33.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:33.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:33.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:33.988 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:33.988+0000 7f7b79e7e7c0 -1 Falling back to public interface 2024-09-17T18:36:34.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:34.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:34.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:36:34.354 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:36:34.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:34.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:34.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:35.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:35.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:35.698 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:36:35.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:36:35.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:35.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:36.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:37.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:37.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:37.010 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:36:37.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:36:37.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:37.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:37.044 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:36:37.044+0000 7f7b79e7e7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:36:37.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:38.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:38.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:38.343 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:36:38.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:36:38.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:38.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:38.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:39.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:39.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:39.765 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:36:39.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:36:39.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:39.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:40.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:41.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:41.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:41.218 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:36:41.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:36:41.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:41.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:41.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:42.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:42.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:42.547 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:36:42.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:36:42.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:42.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:43.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:44.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:44.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:44.037 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:36:44.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:36:44.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:44.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:44.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:45.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:45.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:45.421 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:36:45.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:36:45.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:45.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:45.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:46.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:46.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:46.763 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:36:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:36:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:46.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:47.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:48.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:48.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:48.146 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:36:48.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:36:48.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:48.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:48.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:49.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:49.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:49.550 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:36:49.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:36:49.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:49.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:50.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:51.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:51.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:51.390 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:36:51.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:36:51.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:51.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:51.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:52.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:52.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:52.710 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:36:52.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:36:52.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:52.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:53.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:54.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:54.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:54.238 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:36:54.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:36:54.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:54.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:54.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:55.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:55.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:55.823 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:36:55.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:36:55.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:55.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:56.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:57.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:57.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:57.379 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:36:57.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:36:57.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:57.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:58.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:36:59.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:36:59.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:36:59.008 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:36:59.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:36:59.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:36:59.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:36:59.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:00.801 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:37:00.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:37:00.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:00.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:37:01.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:02.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:02.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:02.356 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:37:02.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:37:02.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:02.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:37:02.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:03.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:03.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:03.816 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T18:37:03.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T18:37:03.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:03.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:37:04.181 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1161609143,v1:127.0.0.1:6811/1161609143] [v2:127.0.0.1:6812/1161609143,v1:127.0.0.1:6813/1161609143] exists,up dc87d921-2456-48ff-ab7b-6ba3909c3779 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:37:04.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:37:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:37:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:37:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:37:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:37:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:37:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6927506a-e68f-41b1-83a9-5916f3c9ac19 2024-09-17T18:37:04.190 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 6927506a-e68f-41b1-83a9-5916f3c9ac19 2024-09-17T18:37:04.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6927506a-e68f-41b1-83a9-5916f3c9ac19' 2024-09-17T18:37:04.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:37:04.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBQzOlmEXGmDBAABhUXJkNadZb1iuen0bLXXQ== 2024-09-17T18:37:04.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBQzOlmEXGmDBAABhUXJkNadZb1iuen0bLXXQ=="}' 2024-09-17T18:37:04.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6927506a-e68f-41b1-83a9-5916f3c9ac19 -i td/osd-scrub-repair/2/new.json 2024-09-17T18:37:04.655 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:37:04.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:37:04.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBQzOlmEXGmDBAABhUXJkNadZb1iuen0bLXXQ== --osd-uuid 6927506a-e68f-41b1-83a9-5916f3c9ac19 2024-09-17T18:37:04.709 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:04.711+0000 7f59c85b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:04.766 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:04.767+0000 7f59c85b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:04.768 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:04.767+0000 7f59c85b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:04.768 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:04.767+0000 7f59c85b87c0 -1 bdev(0x5632c108fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:37:04.769 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:04.767+0000 7f59c85b87c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:37:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:37:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:37:11.899 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:37:11.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:37:11.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:37:12.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:37:12.299 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:37:12.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:37:12.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:37:12.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:37:12.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:37:12.327 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:12.327+0000 7f912f9eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:12.330 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:12.331+0000 7f912f9eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:12.331 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:12.331+0000 7f912f9eb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:12.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:37:12.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:37:12.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:37:12.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:37:12.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:37:12.647 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:37:12.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:12.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:37:12.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:12.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:12.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:13.812 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:13.811+0000 7f912f9eb7c0 -1 Falling back to public interface 2024-09-17T18:37:13.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:13.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:13.990 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:37:13.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:37:13.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:13.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:14.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:15.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:15.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:15.309 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:37:15.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:37:15.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:15.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:15.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:16.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:16.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:16.642 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:37:16.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:37:16.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:16.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:16.766 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:16.767+0000 7f912f9eb7c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:37:17.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:18.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:18.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:18.032 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:37:18.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:37:18.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:18.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:18.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:19.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:19.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:19.487 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:37:19.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:37:19.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:19.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:19.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:20.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:20.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:20.937 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:37:20.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:37:20.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:20.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:21.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:22.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:22.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:22.272 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:37:22.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:37:22.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:22.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:22.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:23.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:23.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:23.714 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:37:23.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:37:23.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:23.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:24.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:25.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:25.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:25.056 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:37:25.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:37:25.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:25.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:25.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:26.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:26.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:26.393 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:37:26.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:37:26.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:26.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:26.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:27.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:27.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:27.752 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:37:27.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:37:27.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:27.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:28.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:29.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:29.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:29.082 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:37:29.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:37:29.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:29.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:29.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:30.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:30.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:30.412 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:37:30.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:37:30.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:30.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:31.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:32.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:32.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:32.008 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:37:32.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:37:32.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:32.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:32.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:33.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:33.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:33.616 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:37:33.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:37:33.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:33.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:34.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:37:35.119 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:37:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:35.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:36.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:36.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:36.882 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:37:36.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:37:36.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:36.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:37.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:38.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:38.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:38.582 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:37:38.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:37:38.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:38.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:39.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:40.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:40.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:40.231 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:37:40.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:37:40.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:40.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:40.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:41.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:41.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:41.558 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:37:41.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:37:41.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:41.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:42.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:43.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:43.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:43.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T18:37:43.290 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T18:37:43.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:43.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:37:43.618 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/627134128,v1:127.0.0.1:6819/627134128] [v2:127.0.0.1:6820/627134128,v1:127.0.0.1:6821/627134128] exists,up 6927506a-e68f-41b1-83a9-5916f3c9ac19 2024-09-17T18:37:43.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:37:43.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:37:43.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:37:43.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-09-17T18:37:43.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-17T18:37:44.101 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'testpool' created 2024-09-17T18:37:44.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:37:45.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-09-17T18:37:45.538 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 1 size to 2 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:37:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:37:45.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:37:45.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:37:45.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:37:45.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:37:45.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:37:45.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:37:45.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:37:45.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:37:45.670 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:37:46.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:37:46.018 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:37:46.018 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:37:46.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:37:46.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:37:46.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:37:46.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-17T18:37:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-17T18:37:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-17T18:37:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:37:46.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:37:46.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672973 2024-09-17T18:37:46.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672973 2024-09-17T18:37:46.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672973' 2024-09-17T18:37:46.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:37:46.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:37:46.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509444 2024-09-17T18:37:46.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509444 2024-09-17T18:37:46.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672973 2-64424509444' 2024-09-17T18:37:46.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:37:46.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-17T18:37:46.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:37:46.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:37:46.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-17T18:37:46.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:37:46.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-17T18:37:46.500 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836503 2024-09-17T18:37:46.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-17T18:37:46.501 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:37:46.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-17T18:37:46.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:37:47.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:37:47.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:37:48.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-17T18:37:48.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:37:49.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T18:37:49.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:37:49.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-17T18:37:49.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:37:49.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672973 2024-09-17T18:37:49.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:37:49.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:37:49.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672973 2024-09-17T18:37:49.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:37:49.495 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672973 2024-09-17T18:37:49.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672973 2024-09-17T18:37:49.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-17T18:37:49.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:37:49.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-17T18:37:49.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:37:49.852 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:37:49.852 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509444 2024-09-17T18:37:49.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:37:49.853 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509444 2024-09-17T18:37:49.853 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:37:49.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509444 2024-09-17T18:37:49.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-09-17T18:37:49.853 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509444 2024-09-17T18:37:49.854 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:37:50.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509444 -lt 64424509444 2024-09-17T18:37:50.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:37:50.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:37:50.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:37:50.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:37:50.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:37:50.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:37:50.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:37:50.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:37:50.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:37:50.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:37:50.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:37:50.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:37:50.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:37:50.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:37:50.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:37:51.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:37:51.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:37:51.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:37:51.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-09-17T18:37:51.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-09-17T18:37:51.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-09-17T18:37:51.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:51.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:51.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:51.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:51.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:51.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:51.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:51.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:51.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:51.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:52.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:52.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:52.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:52.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:52.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:52.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:52.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-09-17T18:37:52.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-09-17T18:37:53.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-17T18:37:53.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:37:53.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:37:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:37:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:37:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:37:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:37:53.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:37:53.387 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:37:53.387 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:37:53.387 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:37:53.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-09-17T18:37:53.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:37:53.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:37:53.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:37:55.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-09-17T18:37:56.890 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:ff7b1f36:::obj1:head# 2024-09-17T18:37:57.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:37:57.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:37:57.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:37:57.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:37:57.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:37:57.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:37:57.432 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:37:57.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:37:57.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:37:57.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:37:57.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:37:57.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:37:57.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:37:57.469 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:57.471+0000 7fa72deb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:57.472 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:57.471+0000 7fa72deb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:57.473 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:57.475+0000 7fa72deb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:37:57.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:57.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:37:58.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:37:58.508 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:37:58.507+0000 7fa72deb57c0 -1 Falling back to public interface 2024-09-17T18:37:59.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:37:59.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:37:59.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:37:59.078 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:37:59.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:37:59.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:37:59.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:00.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:00.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:00.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:38:00.402 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:38:00.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:00.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:38:00.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:01.321 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:01.319+0000 7fa72deb57c0 -1 osd.0 21 log_to_monitors true 2024-09-17T18:38:01.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:01.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:01.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:38:01.713 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:38:01.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:01.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:38:02.122 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3664948482,v1:127.0.0.1:6803/3664948482] [v2:127.0.0.1:6804/3664948482,v1:127.0.0.1:6805/3664948482] exists,up 924eeb24-8b86-46db-9042-24a865d08e6b 2024-09-17T18:38:02.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:38:02.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:38:02.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:38:02.123 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:38:02.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:38:02.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:38:02.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:38:02.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:38:02.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:38:02.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:38:02.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:38:02.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:38:02.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:38:02.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:38:02.567 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:02.567 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:38:02.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:38:02.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:02.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:38:02.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-17T18:38:02.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-17T18:38:02.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-17T18:38:02.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:02.723 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:38:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-17T18:38:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-17T18:38:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672977' 2024-09-17T18:38:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:02.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:38:03.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-17T18:38:03.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-17T18:38:03.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672977 2-64424509448' 2024-09-17T18:38:03.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:03.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-17T18:38:03.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:03.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:38:03.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-17T18:38:03.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:03.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-17T18:38:03.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-17T18:38:03.011 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 107374182402 2024-09-17T18:38:03.012 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:03.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-17T18:38:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:38:04.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:38:04.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:04.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-17T18:38:04.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:04.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-17T18:38:04.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:04.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:38:04.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-17T18:38:04.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:04.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-17T18:38:04.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-17T18:38:04.663 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672977 2024-09-17T18:38:04.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:38:05.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672977 2024-09-17T18:38:05.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:05.009 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-17T18:38:05.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:05.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:38:05.011 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-17T18:38:05.012 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:05.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-17T18:38:05.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-17T18:38:05.013 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509448 2024-09-17T18:38:05.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:38:05.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-17T18:38:05.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:38:05.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:05.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:05.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:38:05.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:38:05.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:38:05.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:38:05.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:38:05.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:38:05.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:38:05.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:38:06.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:38:06.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:38:06.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:06.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:06.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:38:06.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:38:06.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:38:06.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-17T18:38:06.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:38:06.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:38:06.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:38:06.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:38:06.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:38:06.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:38:06.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:38:06.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:38:06.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:38:08.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:38:08.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-09-17T18:38:08.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:38:08.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:38:08.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:38:08.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:38:08.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:38:08.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:38:10.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:38:10.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:38:10.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:38:10.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:38:10.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:38:10.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:38:10.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:38:10.181 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:38:10.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:38:10.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:38:10.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:38:10.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:38:10.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:38:10.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:38:10.220 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:10.219+0000 7efd308907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:10.222 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:10.223+0000 7efd308907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:10.224 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:10.223+0000 7efd308907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:10.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:38:10.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:38:10.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:38:10.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:38:10.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:38:10.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:10.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:38:10.537 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:38:10.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:10.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:10.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:11.496 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:11.494+0000 7efd308907c0 -1 Falling back to public interface 2024-09-17T18:38:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:38:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:12.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:13.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:13.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:13.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:38:13.281 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:38:13.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:13.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:13.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:14.289 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:14.290+0000 7efd308907c0 -1 osd.1 26 log_to_monitors true 2024-09-17T18:38:14.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:14.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:14.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:38:14.606 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:38:14.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:14.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:14.986 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 30 up_thru 25 down_at 27 last_clean_interval [10,26) [v2:127.0.0.1:6810/1664369375,v1:127.0.0.1:6811/1664369375] [v2:127.0.0.1:6812/1664369375,v1:127.0.0.1:6813/1664369375] exists,up dc87d921-2456-48ff-ab7b-6ba3909c3779 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:38:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:38:14.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:38:14.988 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:38:14.988 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:38:14.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:38:14.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:38:14.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:38:15.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:38:15.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:38:15.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:38:15.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:38:15.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:38:15.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:38:15.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:38:15.477 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:15.477 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:38:15.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:38:15.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:15.478 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:38:15.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182405 2024-09-17T18:38:15.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182405 2024-09-17T18:38:15.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405' 2024-09-17T18:38:15.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:15.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:38:15.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-17T18:38:15.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-17T18:38:15.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-128849018882' 2024-09-17T18:38:15.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:15.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:38:15.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-17T18:38:15.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-17T18:38:15.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182405 1-128849018882 2-64424509452' 2024-09-17T18:38:15.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:15.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182405 2024-09-17T18:38:15.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:15.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:38:15.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182405 2024-09-17T18:38:15.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:15.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182405 2024-09-17T18:38:15.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-09-17T18:38:15.986 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 107374182405 2024-09-17T18:38:15.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:16.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182405 -lt 107374182405 2024-09-17T18:38:16.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:16.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018882 2024-09-17T18:38:16.335 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:16.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:38:16.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018882 2024-09-17T18:38:16.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:16.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-17T18:38:16.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-09-17T18:38:16.339 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 128849018882 2024-09-17T18:38:16.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:38:16.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-17T18:38:16.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:16.680 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-17T18:38:16.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:16.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:38:16.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-17T18:38:16.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:16.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-17T18:38:16.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-17T18:38:16.684 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509452 2024-09-17T18:38:16.685 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:38:17.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-17T18:38:17.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:38:17.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:17.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:17.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:38:17.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:38:17.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:38:17.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:38:17.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:38:17.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:38:17.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:38:17.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:38:17.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:38:17.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:38:17.736 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:17.736 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:18.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:38:18.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:38:18.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:38:18.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-09-17T18:38:18.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:38:18.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:38:18.114 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:38:18.114 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:38:18.114 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:38:18.115 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:38:18.115 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:38:18.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:38:18.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:38:18.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:38:18.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-09-17T18:38:18.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:38:18.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:38:18.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:38:18.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:38:19.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:38:19.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-09-17T18:38:21.078 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:104778fc:::obj2:head# 2024-09-17T18:38:21.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:38:21.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:38:21.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:38:21.629 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:38:21.629 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:38:21.629 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:38:21.630 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:38:21.630 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:21.630 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:21.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:38:21.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:38:21.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:38:21.632 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:38:21.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:38:21.632 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:38:21.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:38:21.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:38:21.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:38:21.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:38:21.668 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:21.666+0000 7f36202c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:21.671 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:21.670+0000 7f36202c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:21.672 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:21.670+0000 7f36202c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:21.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:38:22.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:22.960 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:22.958+0000 7f36202c47c0 -1 Falling back to public interface 2024-09-17T18:38:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:38:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:23.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:23.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:38:23.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:24.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:24.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:24.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:38:24.618 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:38:24.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:24.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:38:24.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:25.742 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:25.742+0000 7f36202c47c0 -1 osd.0 31 log_to_monitors true 2024-09-17T18:38:25.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:25.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:25.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:38:25.953 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:38:25.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:25.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:38:26.321 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 35 up_thru 27 down_at 32 last_clean_interval [25,31) [v2:127.0.0.1:6802/339671459,v1:127.0.0.1:6803/339671459] [v2:127.0.0.1:6804/339671459,v1:127.0.0.1:6805/339671459] exists,up 924eeb24-8b86-46db-9042-24a865d08e6b 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:38:26.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:38:26.323 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:38:26.323 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:38:26.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:38:26.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:38:26.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:38:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:38:26.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:38:26.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:38:26.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:38:26.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:38:26.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:38:26.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:38:26.846 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:26.846 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:38:26.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:38:26.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:26.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:38:26.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-17T18:38:26.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-17T18:38:26.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-17T18:38:26.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:26.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:38:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018885 2024-09-17T18:38:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018885 2024-09-17T18:38:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-128849018885' 2024-09-17T18:38:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:27.134 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:38:27.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T18:38:27.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T18:38:27.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-128849018885 2-64424509455' 2024-09-17T18:38:27.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:27.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-17T18:38:27.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:27.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:38:27.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-17T18:38:27.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:27.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-17T18:38:27.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-17T18:38:27.281 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 150323855362 2024-09-17T18:38:27.281 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:27.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 150323855362 2024-09-17T18:38:27.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:38:28.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:38:28.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:28.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-17T18:38:28.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:28.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-128849018885 2024-09-17T18:38:28.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:28.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:38:28.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-128849018885 2024-09-17T18:38:28.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018885 2024-09-17T18:38:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 128849018885' 2024-09-17T18:38:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 128849018885 2024-09-17T18:38:28.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:38:29.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018885 -lt 128849018885 2024-09-17T18:38:29.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:29.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-17T18:38:29.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:29.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:38:29.262 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-17T18:38:29.262 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:29.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T18:38:29.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-17T18:38:29.264 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509455 2024-09-17T18:38:29.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:38:29.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-17T18:38:29.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:38:29.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:29.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:30.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:38:30.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:38:30.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:38:30.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:38:30.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:38:30.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:38:30.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:38:30.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:38:30.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:38:30.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:38:30.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:30.423 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:30.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:38:30.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:38:30.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:38:30.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-09-17T18:38:30.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:38:30.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:38:30.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:38:30.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:38:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:38:31.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:38:31.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:38:31.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:38:31.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:38:31.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:38:31.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:38:31.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:38:31.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-09-17T18:38:31.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:38:31.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:38:31.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:38:31.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:38:31.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:38:31.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-09-17T18:38:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-09-17T18:38:33.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:38:33.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:38:33.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:38:33.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:38:33.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:38:33.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:38:33.513 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:38:33.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:38:33.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:38:33.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:38:33.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:38:33.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:38:33.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:38:33.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:33.554+0000 7f861318c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:33.558 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:33.558+0000 7f861318c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:33.560 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:33.558+0000 7f861318c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:38:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:38:33.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:33.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:34.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:35.039 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:35.038+0000 7f861318c7c0 -1 Falling back to public interface 2024-09-17T18:38:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:38:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:35.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:36.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:36.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:36.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:38:36.548 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:38:36.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:36.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:36.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:38:37.740 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:37.738+0000 7f861318c7c0 -1 osd.1 36 log_to_monitors true 2024-09-17T18:38:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:38:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:38:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:38:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:38:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:38:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 40 up_thru 35 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6810/3824295220,v1:127.0.0.1:6811/3824295220] [v2:127.0.0.1:6812/3824295220,v1:127.0.0.1:6813/3824295220] exists,up dc87d921-2456-48ff-ab7b-6ba3909c3779 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:38:38.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:38:38.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:38:38.246 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:38:38.246 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:38:38.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:38:38.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:38:38.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:38:38.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:38:38.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:38:38.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:38:38.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:38:38.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:38:38.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:38:38.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:38:38.709 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:38.709 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:38:38.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:38:38.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:38.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:38:38.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855365 2024-09-17T18:38:38.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855365 2024-09-17T18:38:38.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365' 2024-09-17T18:38:38.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:38.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:38:39.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691842 2024-09-17T18:38:39.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691842 2024-09-17T18:38:39.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365 1-171798691842' 2024-09-17T18:38:39.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:39.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:38:39.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-17T18:38:39.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-17T18:38:39.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855365 1-171798691842 2-64424509459' 2024-09-17T18:38:39.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:39.284 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855365 2024-09-17T18:38:39.284 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:39.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:38:39.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855365 2024-09-17T18:38:39.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:39.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855365 2024-09-17T18:38:39.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855365' 2024-09-17T18:38:39.287 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 150323855365 2024-09-17T18:38:39.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855364 -lt 150323855365 2024-09-17T18:38:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:38:40.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:38:40.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:40.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855365 -lt 150323855365 2024-09-17T18:38:40.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:40.903 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691842 2024-09-17T18:38:40.904 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:40.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:38:40.906 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691842 2024-09-17T18:38:40.907 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:40.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691842 2024-09-17T18:38:40.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-09-17T18:38:40.908 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 171798691842 2024-09-17T18:38:40.908 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:38:41.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691842 -lt 171798691842 2024-09-17T18:38:41.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:41.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-17T18:38:41.240 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:41.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:38:41.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:41.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-17T18:38:41.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-17T18:38:41.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-17T18:38:41.244 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509459 2024-09-17T18:38:41.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:38:41.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-17T18:38:41.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:38:41.604 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:41.604 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:42.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:38:42.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:38:42.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:38:42.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:38:42.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:38:42.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:38:42.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:38:42.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:38:42.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:38:42.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:38:42.423 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:42.423 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:42.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:38:42.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:38:42.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:38:42.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-09-17T18:38:42.965 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0: { 2024-09-17T18:38:42.965 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-17T18:38:42.965 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:38:42.978 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1: { 2024-09-17T18:38:42.978 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-17T18:38:42.978 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:38:42.991 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2: { 2024-09-17T18:38:42.991 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-09-17T18:38:42.991 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:38:43.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-09-17T18:38:43.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-17T18:38:43.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-17T18:38:43.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2024-09-17T18:38:43.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:38:43.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-09-17T18:38:43.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-09-17T18:38:43.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:38:43.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-17T18:38:43.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:38:43.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2024-09-17T18:38:43.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-09-17T18:38:43.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-09-17T18:38:43.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:38:43.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:38:43.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:38:43.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:38:44.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:44.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-09-17T18:38:44.206 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:38:44.206 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": true, 2024-09-17T18:38:44.206 INFO:tasks.workunit.client.0.smithi105.stdout: "must": false, 2024-09-17T18:38:44.206 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "2024-09-03T18:37:04.210942+0000" 2024-09-17T18:38:44.206 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:38:44.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:44.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-17T18:38:44.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:44.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:38:44.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:38:44.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:38:44.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:38:44.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:38:44.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:38:44.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:38:44.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:38:44.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:37:44.042273+0000 '>' 2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:44.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:38:45.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:38:45.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:38:45.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:38:45.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:38:45.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:38:45.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:38:45.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:38:45.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:37:44.042273+0000 '>' 2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:45.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:38:46.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:38:46.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:38:46.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:38:46.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:38:46.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:38:46.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:38:46.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:38:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:37:44.042273+0000 '>' 2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:38:48.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:38:48.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:38:48.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:38:48.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:38:48.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:38:48.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:38:48.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:38:45.996722+0000 '>' 2024-09-17T18:37:44.042273+0000 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:38:48.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:38:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:38:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:38:48.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:38:48.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:38:48.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:38:48.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:38:48.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:38:48.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:38:48.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:38:48.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:38:48.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:38:48.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:38:48.961 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:48.961 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:38:48.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:38:48.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:48.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:38:49.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855368 2024-09-17T18:38:49.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855368 2024-09-17T18:38:49.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368' 2024-09-17T18:38:49.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:49.130 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:38:49.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691845 2024-09-17T18:38:49.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691845 2024-09-17T18:38:49.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368 1-171798691845' 2024-09-17T18:38:49.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:49.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:38:49.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-17T18:38:49.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-17T18:38:49.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855368 1-171798691845 2-64424509462' 2024-09-17T18:38:49.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:49.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:49.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855368 2024-09-17T18:38:49.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:38:49.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855368 2024-09-17T18:38:49.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:49.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855368 2024-09-17T18:38:49.414 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 150323855368 2024-09-17T18:38:49.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855368' 2024-09-17T18:38:49.415 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:49.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855367 -lt 150323855368 2024-09-17T18:38:49.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:38:50.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:38:50.726 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:51.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855368 -lt 150323855368 2024-09-17T18:38:51.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:51.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691845 2024-09-17T18:38:51.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:51.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:38:51.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691845 2024-09-17T18:38:51.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691845 2024-09-17T18:38:51.045 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 171798691845 2024-09-17T18:38:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-09-17T18:38:51.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:38:51.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691845 -lt 171798691845 2024-09-17T18:38:51.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:51.395 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-17T18:38:51.395 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:51.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:38:51.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-17T18:38:51.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:51.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-17T18:38:51.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-17T18:38:51.398 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509462 2024-09-17T18:38:51.398 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:38:51.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-17T18:38:51.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:38:51.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:51.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:52.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:38:52.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:38:52.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:38:52.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:38:52.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:38:52.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:38:52.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:38:52.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:38:52.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:38:52.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:38:52.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:38:52.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:38:52.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:38:52.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:38:52.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:38:52.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-09-17T18:38:52.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:38:52.906 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:38:53.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:38:53.231 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:38:53.231 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:38:53.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:38:53.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:53.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:38:53.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855370 2024-09-17T18:38:53.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855370 2024-09-17T18:38:53.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370' 2024-09-17T18:38:53.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:53.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:38:53.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691847 2024-09-17T18:38:53.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691847 2024-09-17T18:38:53.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370 1-171798691847' 2024-09-17T18:38:53.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:38:53.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:38:53.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509464 2024-09-17T18:38:53.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509464 2024-09-17T18:38:53.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855370 1-171798691847 2-64424509464' 2024-09-17T18:38:53.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:53.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855370 2024-09-17T18:38:53.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:53.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:38:53.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855370 2024-09-17T18:38:53.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:53.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855370 2024-09-17T18:38:53.684 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 150323855370 2024-09-17T18:38:53.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855370' 2024-09-17T18:38:53.685 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855369 -lt 150323855370 2024-09-17T18:38:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:38:54.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:38:54.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:38:55.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855370 -lt 150323855370 2024-09-17T18:38:55.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:55.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-171798691847 2024-09-17T18:38:55.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:55.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:38:55.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-171798691847 2024-09-17T18:38:55.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:55.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691847 2024-09-17T18:38:55.310 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 171798691847 2024-09-17T18:38:55.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 171798691847' 2024-09-17T18:38:55.310 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:38:55.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691847 -lt 171798691847 2024-09-17T18:38:55.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:38:55.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509464 2024-09-17T18:38:55.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:38:55.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:38:55.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509464 2024-09-17T18:38:55.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:38:55.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509464 2024-09-17T18:38:55.664 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509464 2024-09-17T18:38:55.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-17T18:38:55.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:38:56.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509464 -lt 64424509464 2024-09-17T18:38:56.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-09-17T18:38:56.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-17T18:38:56.385 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:38:56.385 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-09-17T18:38:45.996896+0000 21'10 41:100 [1,0] 1 [1,0] 1 21'10 2024-09-17T18:38:45.996722+0000 21'10 2024-09-17T18:38:45.996722+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:38:45.996722+0000 10 0 2024-09-17T18:38:56.385 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:38:56.385 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:38:56.385 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:38:56.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-09-17T18:38:56.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-09-17T18:38:56.710 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:38:56.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:38:56.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:38:56.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:38:56.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:38:56.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:38:56.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:38:56.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:38:56.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:38:56.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:38:56.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:38:56.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:38:56.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:38:56.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:38:56.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:38:56.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:38:56.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:38:56.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:38:56.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:38:56.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:38:56.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:38:56.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:38:56.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:38:56.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:38:56.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:38:56.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:56.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:56.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:38:56.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:38:56.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:38:56.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:38:56.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:38:56.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:38:56.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:38:56.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:38:56.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:38:56.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:38:56.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:38:56.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:38:56.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:38:56.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:38:56.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:38:56.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:38:56.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:38:56.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:38:56.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:38:56.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:38:56.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:38:56.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:38:56.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:56.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:56.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:38:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:38:56.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:38:56.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:38:57.373 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:38:57.373 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:38:57.373 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:38:57.374 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:38:57.374 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:57.374 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:57.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:57.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:38:57.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:38:57.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:38:57.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:38:57.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:38:57.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:38:57.587 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:38:57.587 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:38:57.587 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:38:57.587 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:38:57.588 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:38:57.588 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:57.588 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:57.588 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:38:57.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:38:57.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:38:57.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:38:57.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:38:57.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:38:57.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:57.705 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:38:57.706 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:38:57.706 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:38:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:38:58.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:38:58.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:38:58.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:38:58.699 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:38:58.699 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:58.699 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:58.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:58.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:38:58.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:38:58.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-17T18:38:58.737 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:38:58.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:38:58.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-17T18:38:58.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:38:58.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:38:58.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e05fbf92-7a06-45c8-9b50-9c9284337d51 2024-09-17T18:38:58.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e05fbf92-7a06-45c8-9b50-9c9284337d51' 2024-09-17T18:38:58.744 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 e05fbf92-7a06-45c8-9b50-9c9284337d51 2024-09-17T18:38:58.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:38:58.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDCzOlmLjWWLRAAqTrobQl3KPYWjJrg8qiPOw== 2024-09-17T18:38:58.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDCzOlmLjWWLRAAqTrobQl3KPYWjJrg8qiPOw=="}' 2024-09-17T18:38:58.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e05fbf92-7a06-45c8-9b50-9c9284337d51 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:38:59.012 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:38:59.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:38:59.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDCzOlmLjWWLRAAqTrobQl3KPYWjJrg8qiPOw== --osd-uuid e05fbf92-7a06-45c8-9b50-9c9284337d51 2024-09-17T18:38:59.061 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:59.062+0000 7feadc5af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:59.064 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:59.062+0000 7feadc5af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:59.065 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:59.066+0000 7feadc5af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:38:59.065 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:59.066+0000 7feadc5af7c0 -1 bdev(0x55e977d99c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:38:59.066 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:38:59.066+0000 7feadc5af7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:39:05.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:39:05.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:39:05.507 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:39:05.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:39:05.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:39:05.901 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:39:05.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:39:05.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:39:05.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:39:05.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:39:05.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:39:05.937 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:05.942+0000 7f92759cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:05.968 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:05.970+0000 7f92759cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:05.969 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:05.974+0000 7f92759cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:06.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:39:06.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:39:06.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:39:06.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:39:06.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:39:06.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:06.252 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:39:06.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:39:06.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:06.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:06.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:07.264 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:07.266+0000 7f92759cd7c0 -1 Falling back to public interface 2024-09-17T18:39:07.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:07.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:07.603 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:39:07.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:39:07.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:07.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:07.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:08.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:08.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:08.927 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:39:08.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:39:08.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:08.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:09.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:10.053 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:10.057+0000 7f92759cd7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:39:10.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:10.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:10.262 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:39:10.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:39:10.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:10.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:10.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:11.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:11.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:11.735 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:39:11.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:39:11.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:11.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:12.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:13.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:13.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:13.056 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:39:13.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:39:13.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:13.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:13.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:14.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:14.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:14.393 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:39:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:39:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:14.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:15.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:15.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:15.859 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:39:15.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:39:15.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:15.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:16.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:17.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:17.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:17.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:39:17.193 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:39:17.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:17.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:17.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:18.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:18.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:18.529 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:39:18.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:39:18.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:18.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:18.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:19.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:19.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:19.856 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:39:19.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:39:19.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:19.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:20.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:21.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:21.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:21.192 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:39:21.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:39:21.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:21.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:21.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:22.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:22.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:22.532 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:39:22.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:39:22.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:22.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:23.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:24.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:24.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:24.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:39:24.390 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:39:24.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:24.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:25.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:26.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:26.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:26.222 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:39:26.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:39:26.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:26.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:26.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:27.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:27.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:27.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:39:27.905 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:39:27.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:27.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:28.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:29.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:29.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:29.260 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:39:29.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:39:29.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:29.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:29.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:30.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:30.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:30.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:39:30.646 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:39:30.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:30.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:31.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:32.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:32.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:32.142 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:39:32.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:39:32.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:32.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:32.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:33.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:33.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:33.695 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:39:33.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:39:33.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:33.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:39:34.032 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/343286482,v1:127.0.0.1:6803/343286482] [v2:127.0.0.1:6804/343286482,v1:127.0.0.1:6805/343286482] exists,up e05fbf92-7a06-45c8-9b50-9c9284337d51 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:39:34.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:39:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:39:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=25a09ec1-182d-4766-8238-8bc7f76a91da 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 25a09ec1-182d-4766-8238-8bc7f76a91da' 2024-09-17T18:39:34.040 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 25a09ec1-182d-4766-8238-8bc7f76a91da 2024-09-17T18:39:34.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:39:34.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDmzOlmETy7AxAAr/V2YrMcmrNwN3wZfoEEmA== 2024-09-17T18:39:34.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDmzOlmETy7AxAAr/V2YrMcmrNwN3wZfoEEmA=="}' 2024-09-17T18:39:34.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 25a09ec1-182d-4766-8238-8bc7f76a91da -i td/osd-scrub-repair/1/new.json 2024-09-17T18:39:34.420 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:39:34.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:39:34.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDmzOlmETy7AxAAr/V2YrMcmrNwN3wZfoEEmA== --osd-uuid 25a09ec1-182d-4766-8238-8bc7f76a91da 2024-09-17T18:39:34.476 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:34.477+0000 7f0b80ba37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:34.505 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:34.509+0000 7f0b80ba37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:34.507 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:34.509+0000 7f0b80ba37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:34.507 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:34.509+0000 7f0b80ba37c0 -1 bdev(0x55f1d4a0dc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:39:34.507 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:34.509+0000 7f0b80ba37c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:39:40.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:39:40.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:39:40.888 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:39:40.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:39:40.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:39:41.297 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:39:41.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:39:41.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:39:41.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:39:41.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:39:41.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:39:41.333 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:41.336+0000 7fa9fe46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:41.367 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:41.368+0000 7fa9fe46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:41.368 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:41.368+0000 7fa9fe46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:39:41.689 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:39:41.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:41.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:42.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:42.404 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:42.404+0000 7fa9fe46b7c0 -1 Falling back to public interface 2024-09-17T18:39:43.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:43.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:43.032 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:39:43.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:39:43.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:43.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:43.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:44.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:44.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:44.359 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:39:44.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:39:44.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:44.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:44.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:45.137 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:39:45.140+0000 7fa9fe46b7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:39:45.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:45.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:45.690 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:39:45.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:39:45.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:45.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:46.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:47.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:47.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:47.086 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:39:47.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:39:47.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:47.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:47.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:48.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:48.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:48.420 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:39:48.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:39:48.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:48.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:48.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:49.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:49.747 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:39:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:39:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:49.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:50.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:51.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:51.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:51.133 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:39:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:39:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:51.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:52.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:52.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:52.473 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:39:52.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:39:52.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:52.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:52.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:53.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:53.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:53.910 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:39:53.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:39:53.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:53.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:54.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:55.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:55.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:55.243 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:39:55.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:39:55.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:55.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:55.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:56.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:56.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:56.616 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:39:56.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:39:56.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:56.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:57.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:58.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:58.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:58.018 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:39:58.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:39:58.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:58.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:39:58.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:39:59.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:39:59.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:39:59.534 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:39:59.535 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:39:59.535 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:39:59.535 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:00.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:01.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:01.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:01.479 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:40:01.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:40:01.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:01.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:02.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:02.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:02.888 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:40:02.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:40:02.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:02.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:03.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:04.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:04.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:04.466 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:40:04.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:40:04.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:04.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:06.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:06.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:06.254 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:40:06.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:40:06.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:06.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:06.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:07.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:07.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:07.836 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:40:07.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:40:07.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:07.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:08.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:09.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:09.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:09.304 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:40:09.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:40:09.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:09.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:40:09.640 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3473250290,v1:127.0.0.1:6811/3473250290] [v2:127.0.0.1:6812/3473250290,v1:127.0.0.1:6813/3473250290] exists,up 25a09ec1-182d-4766-8238-8bc7f76a91da 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:40:09.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:40:09.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:40:09.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:40:09.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:40:09.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-17T18:40:09.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:40:09.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:40:09.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff0eed9a-f533-4e87-bb6e-e3a0241f4ad4 2024-09-17T18:40:09.647 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 ff0eed9a-f533-4e87-bb6e-e3a0241f4ad4 2024-09-17T18:40:09.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ff0eed9a-f533-4e87-bb6e-e3a0241f4ad4' 2024-09-17T18:40:09.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:40:09.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAJzelm3gIMKBAATNQPegVfQu09M9za1a4gbw== 2024-09-17T18:40:09.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAJzelm3gIMKBAATNQPegVfQu09M9za1a4gbw=="}' 2024-09-17T18:40:09.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff0eed9a-f533-4e87-bb6e-e3a0241f4ad4 -i td/osd-scrub-repair/2/new.json 2024-09-17T18:40:10.048 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:40:10.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:40:10.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQAJzelm3gIMKBAATNQPegVfQu09M9za1a4gbw== --osd-uuid ff0eed9a-f533-4e87-bb6e-e3a0241f4ad4 2024-09-17T18:40:10.096 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:10.096+0000 7f085f8947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:10.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:10.140+0000 7f085f8947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:10.143 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:10.144+0000 7f085f8947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:10.143 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:10.144+0000 7f085f8947c0 -1 bdev(0x5646fc1ce800 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:40:10.143 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:10.144+0000 7f085f8947c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:40:17.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:40:17.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:40:17.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:40:17.187 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:40:17.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:40:17.587 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:40:17.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:40:17.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:40:17.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:40:17.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:40:17.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:40:17.622 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:17.623+0000 7fe081c5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:17.624 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:17.623+0000 7fe081c5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:17.626 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:17.627+0000 7fe081c5a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:40:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:40:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:40:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:40:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:40:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:17.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:40:17.903 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:40:17.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:17.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:18.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:18.920 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:18.919+0000 7fe081c5a7c0 -1 Falling back to public interface 2024-09-17T18:40:19.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:19.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:19.234 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:40:19.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:40:19.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:19.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:19.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:20.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:20.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:20.548 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:40:20.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:40:20.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:20.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:20.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:21.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:21.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:21.871 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:40:21.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:40:21.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:21.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:22.058 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:22.059+0000 7fe081c5a7c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:40:22.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:23.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:40:23.283 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:40:23.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:23.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:23.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:24.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:24.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:24.609 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:40:24.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:40:24.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:24.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:24.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:25.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:25.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:25.928 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:40:25.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:40:25.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:25.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:26.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:27.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:27.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:27.409 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:40:27.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:40:27.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:27.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:27.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:28.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:28.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:28.805 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:40:28.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:40:28.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:28.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:29.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:30.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:30.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:30.152 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:40:30.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:40:30.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:30.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:30.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:31.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:31.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:31.505 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:40:31.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:40:31.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:31.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:31.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:32.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:32.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:32.838 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:40:32.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:40:32.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:32.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:33.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:34.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:34.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:34.177 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:40:34.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:40:34.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:34.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:34.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:35.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:35.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:35.574 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:40:35.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:40:35.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:35.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:36.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:37.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:37.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:37.043 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:40:37.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:40:37.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:37.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:37.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:38.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:38.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:38.442 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:40:38.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:40:38.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:38.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:39.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:40.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:40.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:40.158 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:40:40.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:40:40.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:40.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:40.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:41.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:41.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:41.934 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:40:41.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:40:41.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:41.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:42.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:43.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:43.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:43.700 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:40:43.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:40:43.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:43.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:44.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:45.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:45.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:45.445 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:40:45.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:40:45.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:45.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:40:46.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:40:46.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:46.863 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:40:46.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:40:46.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:46.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:40:47.189 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2567337124,v1:127.0.0.1:6819/2567337124] [v2:127.0.0.1:6820/2567337124,v1:127.0.0.1:6821/2567337124] exists,up ff0eed9a-f533-4e87-bb6e-e3a0241f4ad4 2024-09-17T18:40:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:40:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:40:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:40:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-09-17T18:40:47.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-17T18:40:47.774 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'testpool' created 2024-09-17T18:40:47.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:40:48.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-09-17T18:40:49.504 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 1 size to 2 2024-09-17T18:40:49.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-17T18:40:49.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:40:49.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:40:49.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:40:49.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:40:49.526 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:40:49.526 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:40:49.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:40:49.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:40:49.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:40:49.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:40:49.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:40:49.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:40:49.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:40:49.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:40:49.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:40:49.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:40:49.978 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:40:49.978 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:40:49.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:40:49.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:40:49.979 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:40:50.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-17T18:40:50.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-17T18:40:50.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-17T18:40:50.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:40:50.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:40:50.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-17T18:40:50.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-17T18:40:50.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672971' 2024-09-17T18:40:50.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:40:50.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:40:50.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-17T18:40:50.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-17T18:40:50.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672971 2-64424509448' 2024-09-17T18:40:50.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:40:50.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:40:50.575 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-17T18:40:50.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:40:50.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-17T18:40:50.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:40:50.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-17T18:40:50.578 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836501 2024-09-17T18:40:50.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-17T18:40:50.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:40:50.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-17T18:40:50.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:40:51.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:40:51.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:40:52.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-17T18:40:52.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:40:52.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-17T18:40:52.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:40:52.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:40:52.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-17T18:40:52.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:40:52.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-17T18:40:52.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-17T18:40:52.249 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672971 2024-09-17T18:40:52.249 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:40:52.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-17T18:40:52.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:40:52.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-17T18:40:52.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:40:52.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:40:52.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-17T18:40:52.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:40:52.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-17T18:40:52.574 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509448 2024-09-17T18:40:52.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-17T18:40:52.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:40:52.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-17T18:40:52.917 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:40:52.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:40:52.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:40:53.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:40:53.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:40:53.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:40:53.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:40:53.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:40:53.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:40:53.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:40:53.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:40:53.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:40:53.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:40:53.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:40:53.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:40:54.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:40:54.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:40:54.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:40:54.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:40:54.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:40:54.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:40:54.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:40:54.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:40:54.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:40:54.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:40:54.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:40:54.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:40:54.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:40:54.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:40:54.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:40:54.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:40:54.496 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:40:54.496 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:40:54.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:40:54.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T18:40:54.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:40:54.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:40:54.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:40:54.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:40:54.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:40:54.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:40:54.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:40:55.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:40:55.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:40:55.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T18:40:55.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:40:55.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:40:55.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:40:55.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:40:55.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:40:55.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:40:55.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:40:55.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:40:55.437 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:40:55.437 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:40:55.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:40:55.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:40:55.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:40:55.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:40:56.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:40:56.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:40:56.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:40:56.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-09-17T18:40:56.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-09-17T18:40:56.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:40:56.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-17T18:40:56.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:40:56.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:40:56.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:40:56.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:40:56.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:40:56.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:40:56.318 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:40:56.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:40:56.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:40:56.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:40:56.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:40:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:40:56.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:40:56.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:40:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:40:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:40:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:40:57.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:40:57.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:40:57.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:40:57.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:40:57.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:40:57.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:40:57.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:40:57.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-17T18:40:59.022 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-17T18:40:59.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:40:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:40:59.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:40:59.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:40:59.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:40:59.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:40:59.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:40:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:40:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:40:59.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:40:59.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:40:59.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:40:59.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:40:59.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:40:59.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:40:59.599 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:59.598+0000 7fcc9da457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:59.602 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:59.602+0000 7fcc9da457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:59.604 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:40:59.602+0000 7fcc9da457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:40:59.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:40:59.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:00.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:00.897 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:00.894+0000 7fcc9da457c0 -1 Falling back to public interface 2024-09-17T18:41:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:01.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:41:01.251 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:41:01.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:01.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:01.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:02.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:02.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:02.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:41:02.576 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:41:02.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:02.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:02.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:03.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:03.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:03.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:41:03.907 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:41:03.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:03.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:04.006 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:04.006+0000 7fcc9da457c0 -1 osd.0 21 log_to_monitors true 2024-09-17T18:41:04.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:05.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:05.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:05.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:41:05.235 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:41:05.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:05.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3807063601,v1:127.0.0.1:6803/3807063601] [v2:127.0.0.1:6804/3807063601,v1:127.0.0.1:6805/3807063601] exists,up e05fbf92-7a06-45c8-9b50-9c9284337d51 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:41:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:41:05.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:41:05.552 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:41:05.552 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:41:05.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:41:05.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:41:05.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:41:05.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:41:05.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:41:05.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:41:05.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:41:05.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:41:05.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:41:05.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:41:05.975 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:41:05.975 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:41:05.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:41:05.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:05.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:41:06.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-17T18:41:06.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-17T18:41:06.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-17T18:41:06.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:06.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:41:06.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-17T18:41:06.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-17T18:41:06.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672975' 2024-09-17T18:41:06.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:06.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:41:06.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509452 2024-09-17T18:41:06.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509452 2024-09-17T18:41:06.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672975 2-64424509452' 2024-09-17T18:41:06.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:06.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-17T18:41:06.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:06.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:41:06.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-17T18:41:06.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:06.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-17T18:41:06.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-17T18:41:06.430 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 107374182402 2024-09-17T18:41:06.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:41:06.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 107374182402 2024-09-17T18:41:06.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:41:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:41:07.758 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:41:08.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-17T18:41:08.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:08.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-17T18:41:08.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:08.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:41:08.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-17T18:41:08.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:08.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-17T18:41:08.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-17T18:41:08.135 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672975 2024-09-17T18:41:08.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:41:08.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-17T18:41:08.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:08.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509452 2024-09-17T18:41:08.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:08.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:41:08.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509452 2024-09-17T18:41:08.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:08.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509452 2024-09-17T18:41:08.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-17T18:41:08.474 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509452 2024-09-17T18:41:08.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:41:08.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-17T18:41:08.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:41:08.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:08.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:09.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:41:09.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:41:09.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:41:09.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:41:09.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:41:09.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:41:09.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:41:09.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:41:09.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:41:09.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:41:09.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:09.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:09.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:41:09.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:41:09.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:41:09.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-09-17T18:41:09.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-17T18:41:09.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:41:09.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-17T18:41:09.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:41:10.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-09-17T18:41:10.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-09-17T18:41:10.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:41:10.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:41:10.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:41:10.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:41:10.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-09-17T18:41:10.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-09-17T18:41:10.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:41:10.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:41:10.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:41:10.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:41:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-09-17T18:41:11.112 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T18:41:11.112 INFO:tasks.workunit.client.0.smithi105.stdout: "deep": false, 2024-09-17T18:41:11.112 INFO:tasks.workunit.client.0.smithi105.stdout: "must": false, 2024-09-17T18:41:11.112 INFO:tasks.workunit.client.0.smithi105.stdout: "stamp": "2024-09-10T18:39:31.113641+0000" 2024-09-17T18:41:11.112 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.0 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:41:11.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:41:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:40:47.643307+0000 '>' 2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:41:12.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:41:12.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:41:12.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:41:12.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:41:12.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:41:12.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:41:12.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:41:12.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:40:47.643307+0000 '>' 2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:12.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:41:13.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:41:13.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:41:13.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:41:13.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:41:13.753 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:41:13.754 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:41:13.754 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:41:14.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:40:47.643307+0000 '>' 2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:14.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:41:15.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:41:15.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:41:15.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-17T18:41:15.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:41:15.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:41:15.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:41:15.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:41:15.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:41:12.694295+0000 '>' 2024-09-17T18:40:47.643307+0000 2024-09-17T18:41:15.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:41:15.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-17T18:41:15.761 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:41:15.761 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-17T18:41:12.922689+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-17T18:41:12.694295+0000 21'1 2024-09-17T18:41:12.694295+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:41:12.694295+0000 1 0 2024-09-17T18:41:15.761 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:41:15.761 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:41:15.761 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:41:15.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-17T18:41:17.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-17T18:41:18.097 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:41:18.098 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-17T18:41:12.922689+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-17T18:41:12.694295+0000 21'1 2024-09-17T18:41:12.694295+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:41:12.694295+0000 1 0 2024-09-17T18:41:18.098 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:41:18.098 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:41:18.098 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:41:18.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-09-17T18:41:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-17T18:41:20.419 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:41:20.419 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-17T18:41:12.922689+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-17T18:41:12.694295+0000 21'1 2024-09-17T18:41:12.694295+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:41:12.694295+0000 1 0 2024-09-17T18:41:20.419 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:41:20.419 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:41:20.419 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:41:20.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-09-17T18:41:25.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-09-17T18:41:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:41:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:41:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:41:25.437 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:41:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:41:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:41:25.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:41:25.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:41:25.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:41:25.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:41:25.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:41:25.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:41:25.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:41:25.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:41:25.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:41:25.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:41:25.931 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:41:25.931 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:41:25.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:41:25.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:25.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:41:26.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182407 2024-09-17T18:41:26.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182407 2024-09-17T18:41:26.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407' 2024-09-17T18:41:26.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:26.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:41:26.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-17T18:41:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-17T18:41:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672980' 2024-09-17T18:41:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:26.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:41:26.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-17T18:41:26.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-17T18:41:26.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672980 2-64424509457' 2024-09-17T18:41:26.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:26.384 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182407 2024-09-17T18:41:26.384 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:26.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:41:26.386 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182407 2024-09-17T18:41:26.386 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:26.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182407 2024-09-17T18:41:26.387 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 107374182407 2024-09-17T18:41:26.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-09-17T18:41:26.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:41:26.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182407 2024-09-17T18:41:26.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:41:27.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:41:27.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:41:28.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182407 -lt 107374182407 2024-09-17T18:41:28.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:28.059 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-17T18:41:28.059 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:28.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:41:28.064 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-17T18:41:28.064 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:28.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-17T18:41:28.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-17T18:41:28.065 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672980 2024-09-17T18:41:28.065 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:41:28.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-17T18:41:28.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:28.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:28.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-17T18:41:28.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:41:28.399 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-17T18:41:28.399 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:28.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-17T18:41:28.401 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509457 2024-09-17T18:41:28.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-17T18:41:28.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:41:28.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509457 2024-09-17T18:41:28.726 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:41:28.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:28.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:29.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:41:29.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:41:29.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:41:29.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:41:29.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:41:29.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:41:29.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:41:29.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:41:29.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:41:29.450 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:41:29.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:29.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:29.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:41:29.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:41:29.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:41:29.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-09-17T18:41:30.125 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:41:30.125 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-17T18:41:12.922689+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-09-17T18:41:12.694295+0000 21'1 2024-09-17T18:41:12.694295+0000 0 1 periodic scrub scheduled @ 2024-09-18T18:41:12.694295+0000 1 0 2024-09-17T18:41:30.125 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:41:30.126 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:41:30.126 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:41:30.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-09-17T18:41:30.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:41:30.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:41:30.144 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:41:30.144 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:41:30.144 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:41:30.144 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:41:30.144 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:41:30.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:41:30.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:41:30.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:41:30.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:41:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:41:31.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:41:31.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-17T18:41:31.586 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:41:31.586 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:41:31.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:41:31.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:41:31.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:31.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:31.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:31.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:41:31.874 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:41:31.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-09-17T18:41:31.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:41:31.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:41:31.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:41:31.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:41:31.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:41:31.911 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:31.913+0000 7f426f5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:31.929 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:31.929+0000 7f426f5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:31.931 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:31.933+0000 7f426f5777c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:32.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:41:32.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:41:32.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:32.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:32.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:32.937 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:32.937+0000 7f426f5777c0 -1 Falling back to public interface 2024-09-17T18:41:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:41:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:41:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:33.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:34.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:34.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:34.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:41:34.886 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:41:34.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:34.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:36.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:36.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:36.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:41:36.206 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:41:36.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:36.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:36.222 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:36.221+0000 7f426f5777c0 -1 osd.0 26 log_to_monitors true 2024-09-17T18:41:36.732 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3752119769,v1:127.0.0.1:6803/3752119769] [v2:127.0.0.1:6804/3752119769,v1:127.0.0.1:6805/3752119769] exists,up e05fbf92-7a06-45c8-9b50-9c9284337d51 2024-09-17T18:41:36.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:41:36.733 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:41:36.734 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:41:36.734 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:41:36.734 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:41:36.734 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:41:36.734 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:41:36.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:41:36.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:41:36.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:41:36.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:41:36.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:41:36.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:41:37.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:41:37.172 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:41:37.172 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:41:37.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:41:37.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:37.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:41:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-17T18:41:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-17T18:41:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-17T18:41:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:37.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:41:37.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-17T18:41:37.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-17T18:41:37.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672983' 2024-09-17T18:41:37.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:41:37.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:41:37.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-17T18:41:37.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-17T18:41:37.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672983 2-64424509461' 2024-09-17T18:41:37.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:37.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-17T18:41:37.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:37.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:41:37.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-17T18:41:37.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:37.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-17T18:41:37.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-17T18:41:37.760 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 128849018882 2024-09-17T18:41:37.761 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:41:38.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-17T18:41:38.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:38.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-17T18:41:38.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:38.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:41:38.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-17T18:41:38.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:38.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-17T18:41:38.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-17T18:41:38.079 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672983 2024-09-17T18:41:38.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:41:38.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672983 2024-09-17T18:41:38.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:41:39.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:41:39.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:41:39.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-17T18:41:39.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:41:39.740 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-17T18:41:39.741 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:41:39.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:41:39.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-17T18:41:39.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:41:39.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-17T18:41:39.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-17T18:41:39.744 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509461 2024-09-17T18:41:39.745 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:41:40.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509461 2024-09-17T18:41:40.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:41:40.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:40.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:40.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:41:40.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:41:40.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:41:40.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:41:40.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:41:40.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:41:40.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:41:40.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:41:40.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:41:40.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:41:40.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:40.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:41.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:41:41.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:41:41.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:41:41.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:41:41.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:41:41.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:41:41.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:41:41.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:41:41.271 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:41:41.271 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:41:41.271 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:41:41.271 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:41:41.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:41:41.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:41:41.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:41:41.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:41:41.597 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:41:41.597 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:41:41.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:41:41.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:41:41.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:41:41.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:41:42.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:41:42.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-17T18:41:42.007 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:11.666+0000 7fa9e23f5640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-09-17T18:41:42.007 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:11.666+0000 7fa9e23f5640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-09-17T18:41:42.007 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:11.666+0000 7fa9e23f5640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-09-17T18:41:42.007 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:12.694+0000 7fa9e23f5640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] NEED_AUTO] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:41:42.007 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:12.694+0000 7fa9e23f5640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:41:42.007 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:12.694+0000 7fa9e23f5640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-17T18:41:42.008 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:12.694+0000 7fa9e23f5640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-17T18:41:42.008 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:41:12.694+0000 7fa9e23f5640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:41:42.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-09-17T18:41:42.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-09-17T18:41:42.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:41:42.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:41:42.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:41:42.250 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:41:42.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:41:42.251 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:41:42.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:41:42.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:41:42.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:41:42.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:41:42.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:41:42.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:41:42.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:41:42.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:41:42.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:41:42.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:41:42.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:41:42.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:42.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:42.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:41:42.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:41:42.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:41:42.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:41:42.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:41:42.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:41:42.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:41:42.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:41:42.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:41:42.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:41:42.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:41:42.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:41:42.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:41:42.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:41:42.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:41:42.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:41:42.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:41:42.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:41:42.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:41:42.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:41:42.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:41:42.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:41:42.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:41:42.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:42.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:42.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:41:42.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:41:42.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:41:42.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:41:42.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:41:42.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:42.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:42.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:41:42.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:41:42.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:41:42.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:41:42.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:41:42.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:41:42.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:41:42.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:41:42.785 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:41:42.785 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:42.785 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:42.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:42.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:41:42.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:41:42.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:41:42.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:41:42.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:41:42.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:41:42.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:42.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:41:42.999 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:41:42.999 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:41:43.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:41:43.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:41:43.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:41:43.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:41:43.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:41:43.117 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:41:43.117 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:41:43.118 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:41:43.118 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:41:43.118 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:43.118 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:43.118 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:41:43.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:41:43.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:41:43.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:44.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:41:44.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:41:44.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:41:44.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-09-17T18:41:44.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-17T18:41:44.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:41:44.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:41:44.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:41:44.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:41:44.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:41:44.150 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:41:44.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f38ab874-c5e6-46ac-af76-444e886501ae' 2024-09-17T18:41:44.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:41:44.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBozelmWF4RChAAhd7s3p9IfXgqpXaSfBb1Wg== 2024-09-17T18:41:44.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBozelmWF4RChAAhd7s3p9IfXgqpXaSfBb1Wg=="}' 2024-09-17T18:41:44.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f38ab874-c5e6-46ac-af76-444e886501ae -i td/osd-scrub-repair/0/new.json 2024-09-17T18:41:44.398 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:41:44.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:41:44.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBozelmWF4RChAAhd7s3p9IfXgqpXaSfBb1Wg== --osd-uuid f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:41:44.453 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:44.453+0000 7f7fdc8337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:44.455 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:44.457+0000 7f7fdc8337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:44.457 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:44.457+0000 7f7fdc8337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:44.457 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:44.457+0000 7f7fdc8337c0 -1 bdev(0x556796e31c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:41:44.457 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:44.457+0000 7f7fdc8337c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:41:50.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:41:50.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:41:50.726 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:41:50.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:41:50.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:41:51.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:41:51.132 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:41:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:41:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:41:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:41:51.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:41:51.169 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:51.168+0000 7f7fedc1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:51.196 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:51.196+0000 7f7fedc1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:51.197 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:51.196+0000 7f7fedc1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:51.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:41:51.482 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:41:51.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:51.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:51.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:52.498 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:52.496+0000 7f7fedc1d7c0 -1 Falling back to public interface 2024-09-17T18:41:52.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:52.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:52.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:41:52.825 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:41:52.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:52.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:53.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:54.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:54.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:54.152 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:41:54.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:41:54.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:54.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:54.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:55.295 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:55.296+0000 7f7fedc1d7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:41:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:55.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:55.488 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:41:55.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:41:55.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:55.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:55.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:41:56.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:41:56.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:41:56.850 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:41:56.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:41:56.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:41:56.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:41:57.187 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1698942428,v1:127.0.0.1:6803/1698942428] [v2:127.0.0.1:6804/1698942428,v1:127.0.0.1:6805/1698942428] exists,up f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:41:57.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:41:57.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:41:57.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:41:57.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-17T18:41:57.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:41:57.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:41:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:41:57.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:41:57.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:41:57.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:41:57.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:41:57.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:41:57.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:41:57.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:41:57.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:41:57.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:41:57.196 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:41:57.196 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:41:57.196 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:41:57.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:57.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:41:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:41:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:41:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:41:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:41:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:41:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:41:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5f39469e-fd1a-496e-a383-b5bd1e010ddd 2024-09-17T18:41:57.200 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 5f39469e-fd1a-496e-a383-b5bd1e010ddd 2024-09-17T18:41:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5f39469e-fd1a-496e-a383-b5bd1e010ddd' 2024-09-17T18:41:57.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:41:57.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB1zelmaaW6DBAAbZsiTiV6snJirt0+MAQmPw== 2024-09-17T18:41:57.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB1zelmaaW6DBAAbZsiTiV6snJirt0+MAQmPw=="}' 2024-09-17T18:41:57.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5f39469e-fd1a-496e-a383-b5bd1e010ddd -i td/osd-scrub-repair/1/new.json 2024-09-17T18:41:57.608 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:41:57.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:41:57.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQB1zelmaaW6DBAAbZsiTiV6snJirt0+MAQmPw== --osd-uuid 5f39469e-fd1a-496e-a383-b5bd1e010ddd 2024-09-17T18:41:57.667 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:57.668+0000 7f1e656207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:57.709 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:57.708+0000 7f1e656207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:57.711 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:57.712+0000 7f1e656207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:41:57.711 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:57.712+0000 7f1e656207c0 -1 bdev(0x55774894dc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:41:57.712 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:41:57.712+0000 7f1e656207c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:42:04.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:42:04.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:42:04.318 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:42:04.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:42:04.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:42:04.721 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:42:04.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:42:04.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:42:04.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:42:04.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:42:04.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:42:04.757 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:04.756+0000 7f69793467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:04.787 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:04.788+0000 7f69793467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:04.789 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:04.788+0000 7f69793467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:42:05.087 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:42:05.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:05.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:42:05.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:06.069 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:06.068+0000 7f69793467c0 -1 Falling back to public interface 2024-09-17T18:42:06.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:06.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:06.439 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:42:06.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:42:06.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:06.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:42:06.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:07.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:07.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:07.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:42:07.757 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:42:07.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:07.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:42:08.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:09.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:09.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:09.077 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:42:09.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:42:09.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:09.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:42:09.129 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:09.128+0000 7f69793467c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:42:09.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:10.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:10.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:10.454 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:42:10.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:42:10.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:10.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:42:10.786 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/542841225,v1:127.0.0.1:6811/542841225] [v2:127.0.0.1:6812/542841225,v1:127.0.0.1:6813/542841225] exists,up 5f39469e-fd1a-496e-a383-b5bd1e010ddd 2024-09-17T18:42:10.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:42:10.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:42:10.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:42:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:42:10.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:42:10.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:42:10.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ef10684b-1e04-4ce1-bf61-fd351dd549bd 2024-09-17T18:42:10.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ef10684b-1e04-4ce1-bf61-fd351dd549bd' 2024-09-17T18:42:10.792 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 ef10684b-1e04-4ce1-bf61-fd351dd549bd 2024-09-17T18:42:10.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:42:10.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCzelmEbFTMBAAOFnJ6qWCpFADtXCWYrvzIg== 2024-09-17T18:42:10.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCzelmEbFTMBAAOFnJ6qWCpFADtXCWYrvzIg=="}' 2024-09-17T18:42:10.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ef10684b-1e04-4ce1-bf61-fd351dd549bd -i td/osd-scrub-repair/2/new.json 2024-09-17T18:42:11.225 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:42:11.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:42:11.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQCCzelmEbFTMBAAOFnJ6qWCpFADtXCWYrvzIg== --osd-uuid ef10684b-1e04-4ce1-bf61-fd351dd549bd 2024-09-17T18:42:11.284 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:11.284+0000 7f763ff727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:11.335 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:11.336+0000 7f763ff727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:11.336 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:11.336+0000 7f763ff727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:11.337 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:11.336+0000 7f763ff727c0 -1 bdev(0x56345fcd5c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:42:11.337 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:11.336+0000 7f763ff727c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:42:17.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:42:17.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:42:17.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:42:17.532 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:42:17.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:42:17.938 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:42:17.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:42:17.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:42:17.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:42:17.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:42:17.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:42:17.973 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:17.972+0000 7f4279d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:18.006 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:18.004+0000 7f4279d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:18.007 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:18.008+0000 7f4279d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:18.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:42:18.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:42:18.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:42:18.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:42:18.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:42:18.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:18.277 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:42:18.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:42:18.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:18.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:42:18.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:19.533 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:19.532+0000 7f4279d1f7c0 -1 Falling back to public interface 2024-09-17T18:42:19.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:19.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:19.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:42:19.595 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:42:19.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:19.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:42:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:20.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:20.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:20.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:42:20.930 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:42:20.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:20.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:42:21.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:22.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:22.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:22.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:42:22.261 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:42:22.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:22.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:42:22.441 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:22.440+0000 7f4279d1f7c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:42:22.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:23.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:42:23.584 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:42:23.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:23.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:42:23.905 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3076631455,v1:127.0.0.1:6819/3076631455] [v2:127.0.0.1:6820/3076631455,v1:127.0.0.1:6821/3076631455] exists,up ef10684b-1e04-4ce1-bf61-fd351dd549bd 2024-09-17T18:42:23.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:42:23.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:42:23.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:42:23.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-09-17T18:42:23.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2024-09-17T18:42:24.343 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'testpool' created 2024-09-17T18:42:24.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:42:25.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-09-17T18:42:25.957 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 1 size to 2 2024-09-17T18:42:25.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-09-17T18:42:25.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:42:25.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:42:25.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:42:25.979 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:42:25.979 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:42:25.979 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:42:25.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:42:25.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:42:25.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:42:26.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:42:26.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:42:26.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:42:26.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:42:26.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:42:26.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:42:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:42:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:42:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:42:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:42:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:26.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:42:26.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836487 2024-09-17T18:42:26.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836487 2024-09-17T18:42:26.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487' 2024-09-17T18:42:26.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:26.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:42:26.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-17T18:42:26.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-17T18:42:26.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-09-17T18:42:26.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:26.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:42:26.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509442 2024-09-17T18:42:26.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509442 2024-09-17T18:42:26.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509442' 2024-09-17T18:42:26.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:26.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836487 2024-09-17T18:42:26.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:26.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:42:26.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836487 2024-09-17T18:42:26.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:26.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836487 2024-09-17T18:42:26.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-17T18:42:26.937 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836487 2024-09-17T18:42:26.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:27.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-17T18:42:27.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:27.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-17T18:42:27.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:27.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:42:27.297 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-17T18:42:27.297 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:27.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-17T18:42:27.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-17T18:42:27.299 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672965 2024-09-17T18:42:27.299 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:42:27.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-17T18:42:27.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:27.690 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509442 2024-09-17T18:42:27.691 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:27.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:42:27.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509442 2024-09-17T18:42:27.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:27.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509442 2024-09-17T18:42:27.695 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509442 2024-09-17T18:42:27.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-09-17T18:42:27.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:42:27.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509441 -lt 64424509442 2024-09-17T18:42:27.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:42:28.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:42:28.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:42:29.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509442 -lt 64424509442 2024-09-17T18:42:29.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:42:29.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:42:29.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:42:29.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:42:29.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:42:30.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T18:42:30.057 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:42:30.057 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:42:30.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:42:30.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-17T18:42:30.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T18:42:30.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:42:30.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T18:42:30.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:42:30.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:42:30.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:42:30.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:42:30.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:42:30.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:42:30.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:42:30.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:42:30.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:42:30.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:42:30.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:42:30.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:42:30.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:42:30.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:42:31.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:42:31.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:42:31.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:42:31.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-09-17T18:42:31.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-09-17T18:42:31.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:42:31.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-17T18:42:31.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:42:31.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:42:31.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:42:31.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:42:31.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:42:31.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:42:31.468 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:42:31.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:42:31.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:42:31.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:42:32.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:42:32.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:42:32.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:42:32.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:42:32.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:42:32.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:42:32.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:42:32.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:42:32.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T18:42:32.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:42:32.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:42:32.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:42:32.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:42:32.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:42:32.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-17T18:42:34.148 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-17T18:42:34.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:42:34.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:42:34.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:42:34.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:42:34.687 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:42:34.688 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:42:34.689 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:42:34.689 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:42:34.689 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:42:34.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:42:34.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:42:34.690 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:42:34.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:42:34.691 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:42:34.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:42:34.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:42:34.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:42:34.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:42:34.727 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:34.727+0000 7f005f0107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:34.730 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:34.727+0000 7f005f0107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:34.731 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:34.731+0000 7f005f0107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:42:35.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:35.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:42:35.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:36.029 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:36.027+0000 7f005f0107c0 -1 Falling back to public interface 2024-09-17T18:42:36.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:36.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:36.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:42:36.351 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:42:36.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:36.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:42:36.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:37.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:37.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:37.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:42:37.686 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:42:37.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:37.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:42:38.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:42:38.830 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:42:38.827+0000 7f005f0107c0 -1 osd.0 21 log_to_monitors true 2024-09-17T18:42:39.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:42:39.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:42:39.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:42:39.020 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:42:39.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:42:39.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/562034684,v1:127.0.0.1:6803/562034684] [v2:127.0.0.1:6804/562034684,v1:127.0.0.1:6805/562034684] exists,up f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:42:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:42:39.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:42:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:42:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:42:39.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:42:39.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:42:39.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:42:39.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:42:39.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:42:39.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:42:39.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:42:39.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:42:39.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:42:39.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:42:39.801 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:42:39.801 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:42:39.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:42:39.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:39.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:42:39.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182402 2024-09-17T18:42:39.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182402 2024-09-17T18:42:39.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402' 2024-09-17T18:42:39.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:39.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:42:40.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672968 2024-09-17T18:42:40.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672968 2024-09-17T18:42:40.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672968' 2024-09-17T18:42:40.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:40.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:42:40.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509446 2024-09-17T18:42:40.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509446 2024-09-17T18:42:40.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182402 1-42949672968 2-64424509446' 2024-09-17T18:42:40.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:40.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182402 2024-09-17T18:42:40.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:40.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:42:40.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182402 2024-09-17T18:42:40.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:40.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182402 2024-09-17T18:42:40.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-09-17T18:42:40.256 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 107374182402 2024-09-17T18:42:40.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:40.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 107374182402 2024-09-17T18:42:40.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:42:41.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:42:41.670 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:41.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182402 -lt 107374182402 2024-09-17T18:42:41.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:41.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672968 2024-09-17T18:42:41.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:41.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:42:41.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672968 2024-09-17T18:42:41.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:41.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672968 2024-09-17T18:42:41.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-17T18:42:41.998 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672968 2024-09-17T18:42:41.999 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:42:42.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672968 2024-09-17T18:42:42.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:42.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509446 2024-09-17T18:42:42.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:42.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:42:42.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509446 2024-09-17T18:42:42.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:42.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509446 2024-09-17T18:42:42.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-17T18:42:42.340 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509446 2024-09-17T18:42:42.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:42:42.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-17T18:42:42.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:42:42.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:42:42.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:42:43.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:42:43.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:42:43.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:42:43.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:42:43.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:42:43.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:42:43.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:42:43.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:42:43.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:42:43.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:42:43.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:42:43.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:42:43.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:42:43.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:42:43.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:42:43.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-09-17T18:42:43.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2024-09-17T18:42:43.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:42:43.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2024-09-17T18:42:43.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:42:44.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-09-17T18:42:44.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-09-17T18:42:44.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:42:44.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:42:44.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:42:44.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:42:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-09-17T18:42:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-09-17T18:42:44.430 INFO:tasks.workunit.client.0.smithi105.stdout:Affected PG 1.0 w/ primary 1 2024-09-17T18:42:44.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-09-17T18:42:44.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:42:44.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:42:44.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:42:44.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:42:44.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-09-17T18:42:24.299057+0000 2024-09-17T18:42:44.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-09-17T18:42:44.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-09-17T18:42:44.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-09-17T18:42:44.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-09-17T18:42:44.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2024-09-17T18:42:44.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:42:44.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:42:44.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-17T18:42:45.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-09-17T18:42:24.299057+0000 2024-09-17T18:42:45.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-09-17T18:42:45.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T18:42:45.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T18:42:45.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-17T18:42:45.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:42:45.062 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:42:45.063 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T18:42:45.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-09-17T18:42:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T18:42:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-09-17T18:42:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T18:42:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T18:42:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2024-09-17T18:42:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T18:42:45.169 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:42:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T18:42:45.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-09-17T18:42:45.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T18:42:45.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-09-17T18:42:45.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:42:45.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:42:45.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:42:45.597 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:42:45.597 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:42:45.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:42:45.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:45.597 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:42:45.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182404 2024-09-17T18:42:45.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182404 2024-09-17T18:42:45.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182404' 2024-09-17T18:42:45.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:45.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:42:45.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672971 2024-09-17T18:42:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672971 2024-09-17T18:42:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182404 1-42949672971' 2024-09-17T18:42:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:45.903 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:42:46.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-17T18:42:46.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-17T18:42:46.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182404 1-42949672971 2-64424509448' 2024-09-17T18:42:46.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:46.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182404 2024-09-17T18:42:46.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:46.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:42:46.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182404 2024-09-17T18:42:46.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:46.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182404 2024-09-17T18:42:46.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-09-17T18:42:46.047 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 107374182404 2024-09-17T18:42:46.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:46.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182403 -lt 107374182404 2024-09-17T18:42:46.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:42:47.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:42:47.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:47.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182404 -lt 107374182404 2024-09-17T18:42:47.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:47.701 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672971 2024-09-17T18:42:47.701 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:47.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:42:47.705 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672971 2024-09-17T18:42:47.705 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:47.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672971 2024-09-17T18:42:47.706 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672971 2024-09-17T18:42:47.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-09-17T18:42:47.706 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:42:48.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672971 -lt 42949672971 2024-09-17T18:42:48.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:48.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:48.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-17T18:42:48.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:42:48.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-17T18:42:48.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:48.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-17T18:42:48.045 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509448 2024-09-17T18:42:48.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-17T18:42:48.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:42:48.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-17T18:42:48.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-09-17T18:42:48.377 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17 18:42:48.380750821+00:00 2024-09-17T18:42:48.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-09-17T18:42:48.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T18:42:48.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T18:42:48.379 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:42:48.379 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:42:48.379 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:42:48.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T18:42:48.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-09-17T18:42:48.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-09-17T18:42:48.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:48.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:48.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:48.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:48.643 INFO:tasks.workunit.client.0.smithi105.stdout:{"deep":true,"must":false,"stamp":"2024-09-03T18:41:08.481249+0000"}0 ) state now: "active+clean" 2024-09-17T18:42:48.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-09-17T18:42:48.643 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:48.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:48.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:48.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-09-17T18:42:48.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:48.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:48.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:48.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:48.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:49.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:49.103 INFO:tasks.workunit.client.0.smithi105.stdout:1 ) state now: "active+clean" 2024-09-17T18:42:49.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-09-17T18:42:49.104 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-09-17T18:42:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:49.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:49.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:49.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:49.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:49.561 INFO:tasks.workunit.client.0.smithi105.stdout:2 ) state now: "active+clean" 2024-09-17T18:42:49.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-09-17T18:42:49.562 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:49.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:49.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:49.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-09-17T18:42:49.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:49.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:49.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:49.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:49.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:50.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:50.022 INFO:tasks.workunit.client.0.smithi105.stdout:3 ) state now: "active+clean" 2024-09-17T18:42:50.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2024-09-17T18:42:50.023 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:50.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:50.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:50.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-09-17T18:42:50.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:50.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:50.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:50.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:50.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:50.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:50.480 INFO:tasks.workunit.client.0.smithi105.stdout:4 ) state now: "active+clean" 2024-09-17T18:42:50.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2024-09-17T18:42:50.480 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:50.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:50.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:50.481 INFO:tasks.workunit.client.0.smithi105.stdout:loop --------> 4 2024-09-17T18:42:50.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 4 == 4 ']' 2024-09-17T18:42:50.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: initiate_and_fetch_state: echo 'loop --------> ' 4 2024-09-17T18:42:50.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:50.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:50.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:50.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:50.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:50.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stdout:5 ) state now: "active+clean" 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 5 == 4 ']' 2024-09-17T18:42:50.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:51.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:51.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:51.262 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:51.262 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:51.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:51.418 INFO:tasks.workunit.client.0.smithi105.stdout:6 ) state now: "active+clean" 2024-09-17T18:42:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2024-09-17T18:42:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:51.418 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 6 == 4 ']' 2024-09-17T18:42:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:51.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:51.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:51.720 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:51.720 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:51.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:51.878 INFO:tasks.workunit.client.0.smithi105.stdout:7 ) state now: "active+clean" 2024-09-17T18:42:51.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2024-09-17T18:42:51.879 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:51.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:51.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:51.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 7 == 4 ']' 2024-09-17T18:42:51.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:52.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:52.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:52.180 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:52.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:52.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:52.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean"' 2024-09-17T18:42:52.336 INFO:tasks.workunit.client.0.smithi105.stdout:8 ) state now: "active+clean" 2024-09-17T18:42:52.337 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:52.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:52.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:52.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 8 == 4 ']' 2024-09-17T18:42:52.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:52.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:52.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:52.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:52.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:52.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:52.797 INFO:tasks.workunit.client.0.smithi105.stdout:9 ) state now: "active+clean" 2024-09-17T18:42:52.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 9 ') state now: ' '"active+clean"' 2024-09-17T18:42:52.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:52.798 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:52.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:52.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 9 == 4 ']' 2024-09-17T18:42:52.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:53.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:53.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:53.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:53.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:53.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:53.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 10 ') state now: ' '"active+clean"' 2024-09-17T18:42:53.269 INFO:tasks.workunit.client.0.smithi105.stdout:10 ) state now: "active+clean" 2024-09-17T18:42:53.269 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:53.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:53.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:53.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-09-17T18:42:53.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:53.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:53.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:53.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:53.571 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:53.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:53.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 11 ') state now: ' '"active+clean"' 2024-09-17T18:42:53.728 INFO:tasks.workunit.client.0.smithi105.stdout:11 ) state now: "active+clean" 2024-09-17T18:42:53.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:53.729 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:53.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:53.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-09-17T18:42:53.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:54.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:54.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:54.030 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:54.030 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:54.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:54.188 INFO:tasks.workunit.client.0.smithi105.stdout:12 ) state now: "active+clean" 2024-09-17T18:42:54.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 12 ') state now: ' '"active+clean"' 2024-09-17T18:42:54.189 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:54.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:54.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:54.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-09-17T18:42:54.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:54.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:54.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:54.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:54.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:54.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-09-17T18:42:54.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 13 ') state now: ' '"active+clean"' 2024-09-17T18:42:54.649 INFO:tasks.workunit.client.0.smithi105.stdout:13 ) state now: "active+clean" 2024-09-17T18:42:54.649 INFO:tasks.workunit.client.0.smithi105.stdout:"active+clean" 2024-09-17T18:42:54.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:54.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-09-17T18:42:54.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 3 == 4 ']' 2024-09-17T18:42:54.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-09-17T18:42:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-09-17T18:42:54.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-09-17T18:42:54.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-09-17T18:42:54.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-09-17T18:42:55.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-09-17T18:42:55.111 INFO:tasks.workunit.client.0.smithi105.stdout:14 ) state now: "active+clean+scrubbing+deep" 2024-09-17T18:42:55.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 14 ') state now: ' '"active+clean+scrubbing+deep"' 2024-09-17T18:42:55.112 INFO:tasks.workunit.client.0.smithi105.stdout:found scrub 2024-09-17T18:42:55.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-09-17T18:42:55.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-09-17T18:42:55.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-09-17T18:42:55.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-09-17T18:42:55.113 INFO:tasks.workunit.client.0.smithi105.stdout:initiate_and_fetch_state ret: 0 2024-09-17T18:42:55.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-09-17T18:42:55.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.td/osd-scrub-repair.asok 2024-09-17T18:42:55.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-09-17T18:42:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-09-17T18:42:55.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-17T18:42:55.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-17T18:42:55.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-17T18:42:55.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-17T18:42:55.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-17T18:42:55.354 INFO:tasks.workunit.client.0.smithi105.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-17T18:42:55.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-17T18:42:55.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-09-17T18:42:55.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-17T18:42:55.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-17T18:42:55.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-17T18:42:55.656 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-17T18:42:55.656 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-17T18:42:55.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-17T18:42:55.814 INFO:tasks.workunit.client.0.smithi105.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-17T18:42:55.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-17T18:42:55.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-17T18:42:55.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-09-17T18:42:55.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-17T18:42:56.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-17T18:42:56.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-17T18:42:56.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-17T18:42:56.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-17T18:42:56.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-17T18:42:56.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-17T18:42:56.277 INFO:tasks.workunit.client.0.smithi105.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-17T18:42:56.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-17T18:42:56.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-09-17T18:42:56.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-17T18:42:56.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-17T18:42:56.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-17T18:42:56.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-17T18:42:56.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-17T18:42:56.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-17T18:42:56.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-17T18:42:56.740 INFO:tasks.workunit.client.0.smithi105.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-17T18:42:56.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-17T18:42:56.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-09-17T18:42:56.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-17T18:42:57.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-17T18:42:57.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-17T18:42:57.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-17T18:42:57.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-17T18:42:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-09-17T18:42:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-09-17T18:42:57.200 INFO:tasks.workunit.client.0.smithi105.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-09-17T18:42:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-09-17T18:42:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-09-17T18:42:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-09-17T18:42:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:42:57.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:42:57.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:42:57.523 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:42:57.523 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:42:57.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:42:57.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:57.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:42:57.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182407 2024-09-17T18:42:57.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182407 2024-09-17T18:42:57.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407' 2024-09-17T18:42:57.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:57.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:42:57.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-17T18:42:57.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-17T18:42:57.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672974' 2024-09-17T18:42:57.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:42:57.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:42:57.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-17T18:42:57.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-17T18:42:57.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-107374182407 1-42949672974 2-64424509451' 2024-09-17T18:42:57.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:57.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-107374182407 2024-09-17T18:42:57.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:57.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:42:57.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-107374182407 2024-09-17T18:42:57.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:57.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182407 2024-09-17T18:42:57.986 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 107374182407 2024-09-17T18:42:57.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-09-17T18:42:57.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:58.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182406 -lt 107374182407 2024-09-17T18:42:58.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:42:59.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:42:59.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:42:59.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182407 -lt 107374182407 2024-09-17T18:42:59.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:59.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-17T18:42:59.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:59.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:42:59.634 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-17T18:42:59.634 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:59.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-17T18:42:59.636 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672974 2024-09-17T18:42:59.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-17T18:42:59.636 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:42:59.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-17T18:42:59.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:42:59.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-17T18:42:59.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:42:59.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:42:59.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-17T18:42:59.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:42:59.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-17T18:42:59.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-17T18:42:59.986 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509451 2024-09-17T18:42:59.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:43:00.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-17T18:43:00.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-09-17T18:43:00.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-09-17T18:43:00.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-09-17T18:43:00.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-09-17T18:43:00.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-09-17T18:43:00.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-09-17T18:43:00.763 INFO:tasks.workunit.client.0.smithi105.stdout:wait-scrub-end state now: "active+clean" 2024-09-17T18:43:00.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-09-17T18:43:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-17T18:43:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-09-17T18:43:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-09-17T18:43:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-09-17T18:43:00.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-09-17T18:43:01.049 INFO:tasks.workunit.client.0.smithi105.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-17T18:43:01.049 INFO:tasks.workunit.client.0.smithi105.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-09-17T18:42:58.170637+0000 21'1 26:54 [1,0] 1 [1,0] 1 21'1 2024-09-17T18:42:57.862071+0000 21'1 2024-09-17T18:42:57.862071+0000 0 4 periodic scrub scheduled @ 2024-09-18T18:42:57.862071+0000 1 0 2024-09-17T18:43:01.050 INFO:tasks.workunit.client.0.smithi105.stdout: 2024-09-17T18:43:01.050 INFO:tasks.workunit.client.0.smithi105.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-17T18:43:01.050 INFO:tasks.workunit.client.0.smithi105.stderr:dumped pgs 2024-09-17T18:43:01.065 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-17T18:43:01.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:43:01.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:43:01.066 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:43:01.066 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:43:01.066 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:43:01.067 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:43:01.067 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:43:01.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:43:01.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:43:01.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:43:01.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:43:01.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:43:01.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:43:01.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:43:02.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:43:02.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-09-17T18:43:02.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:43:02.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:43:02.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:43:02.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:43:02.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:43:02.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-09-17T18:43:02.532 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:43:02.533 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:43:02.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:43:02.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:43:02.828 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:43:02.828 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:43:02.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:43:02.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:43:02.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:43:02.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:43:02.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:43:02.830 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:43:02.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:43:02.831 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:43:02.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:43:02.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:43:02.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:43:02.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:43:02.867 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:02.867+0000 7f1d7b7b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:02.870 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:02.867+0000 7f1d7b7b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:02.872 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:02.871+0000 7f1d7b7b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:03.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:03.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:04.166 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:04.163+0000 7f1d7b7b87c0 -1 Falling back to public interface 2024-09-17T18:43:04.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:04.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:04.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:43:04.503 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:43:04.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:04.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:04.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:05.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:05.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:05.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:43:05.826 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:43:05.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:05.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:06.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:07.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:07.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:07.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:43:07.164 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:43:07.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:07.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:07.256 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:07.255+0000 7f1d7b7b87c0 -1 osd.0 26 log_to_monitors true 2024-09-17T18:43:07.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:08.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:08.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:08.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:43:08.532 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:43:08.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:08.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:08.876 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3852877411,v1:127.0.0.1:6803/3852877411] [v2:127.0.0.1:6804/3852877411,v1:127.0.0.1:6805/3852877411] exists,up f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:43:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:43:08.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:43:08.878 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:43:08.878 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:43:08.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:43:08.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:43:08.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:43:08.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:43:08.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:43:08.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:43:08.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:43:08.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:43:08.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:43:09.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:43:09.356 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:43:09.357 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:43:09.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:43:09.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:43:09.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:43:09.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018882 2024-09-17T18:43:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018882 2024-09-17T18:43:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882' 2024-09-17T18:43:09.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:43:09.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:43:09.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-17T18:43:09.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-17T18:43:09.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672977' 2024-09-17T18:43:09.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:43:09.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:43:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T18:43:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T18:43:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-128849018882 1-42949672977 2-64424509455' 2024-09-17T18:43:09.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:43:09.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-128849018882 2024-09-17T18:43:09.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:43:09.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:43:09.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-128849018882 2024-09-17T18:43:09.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:43:09.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018882 2024-09-17T18:43:09.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-09-17T18:43:09.795 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 128849018882 2024-09-17T18:43:09.795 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:43:10.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 128849018882 2024-09-17T18:43:10.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:43:11.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:43:11.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:43:11.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018882 -lt 128849018882 2024-09-17T18:43:11.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:43:11.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-17T18:43:11.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:43:11.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:43:11.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-17T18:43:11.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:43:11.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-17T18:43:11.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-17T18:43:11.455 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672977 2024-09-17T18:43:11.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:43:11.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672977 2024-09-17T18:43:11.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:43:11.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-17T18:43:11.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:43:11.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:43:11.795 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-17T18:43:11.795 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:43:11.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T18:43:11.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-17T18:43:11.796 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509455 2024-09-17T18:43:11.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:43:12.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-17T18:43:12.118 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:43:12.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:43:12.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:43:12.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:43:12.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:43:12.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:43:12.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:43:12.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:43:12.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:43:12.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:43:12.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:43:12.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:43:12.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:43:12.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:43:12.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:43:13.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:43:13.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:43:13.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:43:13.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-09-17T18:43:13.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2024-09-17T18:43:13.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:43:13.258 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2024-09-17T18:43:13.258 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2024-09-17T18:43:13.259 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:43:13.259 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:43:13.259 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:43:13.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T18:43:13.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-09-17T18:43:13.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:43:13.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:43:13.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:43:13.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:43:13.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:43:14.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:43:14.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:43:14.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:43:14.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:43:14.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:43:14.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:43:14.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:43:14.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:43:14.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:43:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-09-17T18:43:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:43:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T18:43:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T18:43:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-09-17T18:43:14.971 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T18:43:14.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T18:43:14.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:43:14.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:43:14.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:43:15.014 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:15.010+0000 7f52e4e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:15.017 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:15.014+0000 7f52e4e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:15.018 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:15.014+0000 7f52e4e047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:15.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T18:43:15.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:15.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:15.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:16.042 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:16.038+0000 7f52e4e047c0 -1 Falling back to public interface 2024-09-17T18:43:16.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:16.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:16.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:43:16.678 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:43:16.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:16.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:17.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:43:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:43:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:18.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:19.078 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:19.074+0000 7f52e4e047c0 -1 osd.0 31 log_to_monitors true 2024-09-17T18:43:19.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:19.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:19.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:43:19.354 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:43:19.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:19.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/1724342517,v1:127.0.0.1:6803/1724342517] [v2:127.0.0.1:6804/1724342517,v1:127.0.0.1:6805/1724342517] exists,up f38ab874-c5e6-46ac-af76-444e886501ae 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:43:19.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:43:19.683 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:43:19.683 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:43:19.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:43:19.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:43:19.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:43:19.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:43:19.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:43:19.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:43:19.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:43:19.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:43:19.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:43:20.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:43:20.103 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:43:20.103 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:43:20.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:43:20.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:43:20.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:43:20.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855362 2024-09-17T18:43:20.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855362 2024-09-17T18:43:20.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362' 2024-09-17T18:43:20.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:43:20.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:43:20.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-17T18:43:20.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-17T18:43:20.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672980' 2024-09-17T18:43:20.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:43:20.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:43:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-17T18:43:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-17T18:43:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-150323855362 1-42949672980 2-64424509458' 2024-09-17T18:43:20.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:43:20.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-150323855362 2024-09-17T18:43:20.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:43:20.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:43:20.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-150323855362 2024-09-17T18:43:20.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:43:20.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855362 2024-09-17T18:43:20.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-09-17T18:43:20.532 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 150323855362 2024-09-17T18:43:20.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:43:20.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018883 -lt 150323855362 2024-09-17T18:43:20.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:43:21.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:43:21.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:43:22.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855362 -lt 150323855362 2024-09-17T18:43:22.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:43:22.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-17T18:43:22.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:43:22.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:43:22.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-17T18:43:22.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:43:22.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-17T18:43:22.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-17T18:43:22.154 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672980 2024-09-17T18:43:22.154 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:43:22.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672980 2024-09-17T18:43:22.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:43:22.478 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-17T18:43:22.479 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:43:22.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:43:22.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-17T18:43:22.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:43:22.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-17T18:43:22.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-17T18:43:22.482 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509458 2024-09-17T18:43:22.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:43:22.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-17T18:43:22.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:43:22.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:43:22.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:43:23.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-17T18:43:23.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:43:23.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:43:23.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:43:23.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:43:23.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:43:23.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:43:23.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:43:23.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:43:23.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:43:23.610 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:43:23.610 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:43:23.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-17T18:43:23.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:43:23.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:43:23.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:43:23.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-09-17T18:43:23.990 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:42:57.855+0000 7f695fad5640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-09-17T18:43:23.990 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:42:57.855+0000 7f695fad5640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-09-17T18:43:23.990 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:42:57.855+0000 7f695fad5640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-09-17T18:43:23.990 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:42:57.855+0000 7f695fad5640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-09-17T18:43:23.990 INFO:tasks.workunit.client.0.smithi105.stdout:2024-09-17T18:42:57.855+0000 7f695fad5640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-09-17T18:43:23.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:43:23.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:43:23.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:43:23.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:43:23.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:43:23.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:43:23.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:43:23.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:43:23.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:43:24.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:43:24.303 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:43:24.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:43:24.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:43:24.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:43:24.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:43:24.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:43:24.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:43:24.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:43:24.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:43:24.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:43:24.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:43:24.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:43:24.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:43:24.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:43:24.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:24.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:24.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:43:24.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:43:24.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:43:24.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:43:24.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:43:24.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:43:24.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:43:24.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:43:24.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:43:24.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:43:24.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:43:24.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:43:24.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:43:24.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:43:24.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:43:24.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:43:24.335 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:43:24.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:43:24.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:43:24.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:43:24.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:43:24.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:24.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:24.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:43:24.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:43:24.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:43:24.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:43:24.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:43:24.342 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:24.342 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:24.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:43:24.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:43:24.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:43:24.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:43:24.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:43:24.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:43:24.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:43:24.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:43:24.824 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:43:24.824 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:24.825 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:24.825 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:24.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:43:25.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:43:25.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:43:25.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:43:25.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:43:25.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:43:25.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:43:25.037 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:43:25.037 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:43:25.037 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:43:25.038 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:43:25.038 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:25.038 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:25.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:43:25.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:43:25.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:43:25.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:43:25.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:43:25.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:43:25.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:43:25.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:43:25.150 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:43:25.150 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:43:25.150 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:43:25.150 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:43:25.151 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:25.151 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:25.151 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:43:25.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:43:25.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:43:25.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:26.152 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:26.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:43:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:43:26.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:43:26.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-17T18:43:26.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:43:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:43:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:43:26.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:43:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:43:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:43:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:43:26.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:43:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d65619bb-24b5-4c6e-a214-221b0d504f15 2024-09-17T18:43:26.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d65619bb-24b5-4c6e-a214-221b0d504f15' 2024-09-17T18:43:26.197 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 d65619bb-24b5-4c6e-a214-221b0d504f15 2024-09-17T18:43:26.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:43:26.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDOzelm0v28DBAAf2XujU//Vr1cPiEa+pRztQ== 2024-09-17T18:43:26.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDOzelm0v28DBAAf2XujU//Vr1cPiEa+pRztQ=="}' 2024-09-17T18:43:26.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d65619bb-24b5-4c6e-a214-221b0d504f15 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:43:26.596 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:43:26.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:43:26.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDOzelm0v28DBAAf2XujU//Vr1cPiEa+pRztQ== --osd-uuid d65619bb-24b5-4c6e-a214-221b0d504f15 2024-09-17T18:43:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:26.642+0000 7ff6cc4ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:26.646 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:26.642+0000 7ff6cc4ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:26.646+0000 7ff6cc4ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:26.646+0000 7ff6cc4ee7c0 -1 bdev(0x55b549fa1c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:43:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:26.646+0000 7ff6cc4ee7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:43:32.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:43:32.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:43:32.627 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:43:32.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:43:32.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:43:33.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:43:33.027 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:43:33.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:43:33.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:43:33.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:43:33.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:43:33.065 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:33.062+0000 7fd44c5bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:33.086 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:33.082+0000 7fd44c5bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:33.088 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:33.086+0000 7fd44c5bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:33.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:43:33.327 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:43:33.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:33.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:33.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:34.626 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:34.622+0000 7fd44c5bd7c0 -1 Falling back to public interface 2024-09-17T18:43:34.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:34.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:34.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:43:34.691 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:43:34.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:34.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:35.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:36.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:36.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:36.021 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:43:36.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:43:36.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:36.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:36.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:37.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:37.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:37.351 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:43:37.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:43:37.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:37.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:37.545 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:43:37.542+0000 7fd44c5bd7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:43:37.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:38.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:38.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:38.684 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:43:38.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:43:38.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:38.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:39.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:40.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:40.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:40.003 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:43:40.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:43:40.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:40.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:40.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:41.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:41.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:41.498 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:43:41.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:43:41.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:41.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:41.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:42.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:42.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:42.832 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:43:42.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:43:42.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:42.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:43.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:44.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:44.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:44.227 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:43:44.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:43:44.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:44.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:44.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:45.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:45.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:45.632 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:43:45.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:43:45.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:45.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:45.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:46.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:46.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:46.969 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:43:46.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:43:46.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:46.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:47.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:48.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:48.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:48.457 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:43:48.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:43:48.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:48.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:48.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:49.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:49.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:49.793 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:43:49.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:43:49.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:49.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:50.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:51.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:51.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:51.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:43:51.118 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:43:51.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:51.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:51.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:52.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:52.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:52.899 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:43:52.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:43:52.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:52.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:53.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:54.873 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:43:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:43:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:55.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:56.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:56.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:56.338 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:43:56.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:43:56.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:56.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:56.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:57.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:57.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:57.873 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:43:57.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:43:57.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:57.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:58.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:43:59.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:43:59.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:43:59.196 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:43:59.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:43:59.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:43:59.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:43:59.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:00.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:00.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:00.885 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:44:00.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:44:00.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:00.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:44:01.223 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3601810013,v1:127.0.0.1:6803/3601810013] [v2:127.0.0.1:6804/3601810013,v1:127.0.0.1:6805/3601810013] exists,up d65619bb-24b5-4c6e-a214-221b0d504f15 2024-09-17T18:44:01.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:44:01.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:44:01.228 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:44:01.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:44:01.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:44:01.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:44:01.230 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:44:01.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 2024-09-17T18:44:01.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4' 2024-09-17T18:44:01.231 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 2024-09-17T18:44:01.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:44:01.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxzelm79wXDxAAstePY4irCG9jTE/q4LbA8w== 2024-09-17T18:44:01.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxzelm79wXDxAAstePY4irCG9jTE/q4LbA8w=="}' 2024-09-17T18:44:01.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 -i td/osd-scrub-repair/1/new.json 2024-09-17T18:44:01.701 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:44:01.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:44:01.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDxzelm79wXDxAAstePY4irCG9jTE/q4LbA8w== --osd-uuid 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 2024-09-17T18:44:01.757 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:01.757+0000 7f1a6fdb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:01.786 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:01.785+0000 7f1a6fdb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:01.787 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:01.789+0000 7f1a6fdb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:01.788 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:01.789+0000 7f1a6fdb47c0 -1 bdev(0x561b04e53c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:44:01.788 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:01.789+0000 7f1a6fdb47c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:44:08.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:44:08.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:44:08.508 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:44:08.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:44:08.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:44:08.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:44:08.915 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:44:08.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:44:08.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:44:08.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:44:08.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:44:08.950 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:08.949+0000 7f078bc3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:08.982 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:08.981+0000 7f078bc3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:08.983 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:08.985+0000 7f078bc3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:09.291 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:44:09.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:44:09.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:09.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:09.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:10.022 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:10.021+0000 7f078bc3a7c0 -1 Falling back to public interface 2024-09-17T18:44:10.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:10.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:10.653 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:44:10.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:44:10.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:10.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:11.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:12.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:12.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:12.006 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:44:12.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:44:12.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:12.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:12.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:12.808 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:12.809+0000 7f078bc3a7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:44:13.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:13.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:13.339 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:44:13.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:44:13.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:13.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:13.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:14.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:14.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:14.675 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:44:14.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:44:14.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:14.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:15.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:16.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:16.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:16.011 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:44:16.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:44:16.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:16.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:16.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:17.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:17.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:17.344 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:44:17.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:44:17.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:17.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:17.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:18.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:18.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:18.674 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:44:18.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:44:18.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:18.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:18.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:19.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:19.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:19.991 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:44:19.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:44:19.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:19.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:20.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:21.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:21.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:21.326 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:44:21.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:44:21.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:21.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:21.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:22.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:22.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:22.812 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:44:22.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:44:22.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:22.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:23.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:24.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:24.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:24.267 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:44:24.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:44:24.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:24.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:24.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:25.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:25.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:25.592 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:44:25.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:44:25.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:25.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:26.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:26.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:26.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:44:26.927 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:44:26.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:26.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:27.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:28.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:28.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:28.640 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:44:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:44:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:29.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:30.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:30.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:30.369 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:44:30.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:44:30.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:30.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:30.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:31.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:31.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:31.992 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:44:31.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:44:31.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:31.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:32.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:33.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:33.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:33.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:44:33.665 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:44:33.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:33.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:34.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:35.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:35.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:35.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:44:35.510 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:44:35.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:35.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:35.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:36.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:36.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:36.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:44:36.830 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:44:36.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:36.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:37.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:38.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:38.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:38.509 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:44:38.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:44:38.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:38.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:39.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:40.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:40.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:40.313 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T18:44:40.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T18:44:40.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:40.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:44:40.642 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1528365239,v1:127.0.0.1:6811/1528365239] [v2:127.0.0.1:6812/1528365239,v1:127.0.0.1:6813/1528365239] exists,up 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:44:40.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:44:40.647 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:44:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:44:40.649 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 a34ae5bb-24d2-4d50-9051-39b713d422b1 2024-09-17T18:44:40.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a34ae5bb-24d2-4d50-9051-39b713d422b1 2024-09-17T18:44:40.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 a34ae5bb-24d2-4d50-9051-39b713d422b1' 2024-09-17T18:44:40.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:44:40.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAYzulm2UbUJxAACWpBjMNIiNsnQYvDoH2cdg== 2024-09-17T18:44:40.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAYzulm2UbUJxAACWpBjMNIiNsnQYvDoH2cdg=="}' 2024-09-17T18:44:40.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a34ae5bb-24d2-4d50-9051-39b713d422b1 -i td/osd-scrub-repair/2/new.json 2024-09-17T18:44:41.035 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:44:41.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:44:41.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAYzulm2UbUJxAACWpBjMNIiNsnQYvDoH2cdg== --osd-uuid a34ae5bb-24d2-4d50-9051-39b713d422b1 2024-09-17T18:44:41.094 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:41.092+0000 7f3f653497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:41.162 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:41.160+0000 7f3f653497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:41.163 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:41.164+0000 7f3f653497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:41.164 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:41.164+0000 7f3f653497c0 -1 bdev(0x55a23f6c1c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:44:41.164 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:41.164+0000 7f3f653497c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:44:48.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:44:48.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:44:48.187 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:44:48.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:44:48.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:44:48.597 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:44:48.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:44:48.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:44:48.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:44:48.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:44:48.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:44:48.634 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:48.632+0000 7feb108b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:48.653 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:48.652+0000 7feb108b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:48.655 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:48.652+0000 7feb108b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:48.945 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:44:48.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:44:48.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:48.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:49.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:49.946 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:49.944+0000 7feb108b27c0 -1 Falling back to public interface 2024-09-17T18:44:50.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:50.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:50.305 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:44:50.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:44:50.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:50.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:50.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:51.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:51.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:51.660 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:44:51.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:44:51.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:51.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:51.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:52.657 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:44:52.656+0000 7feb108b27c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:44:53.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:53.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:53.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:44:53.000 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:44:53.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:53.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:53.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:54.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:54.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:54.360 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:44:54.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:44:54.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:54.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:54.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:55.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:55.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:55.698 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:44:55.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:44:55.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:55.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:56.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:57.199 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:44:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:44:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:57.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:58.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:58.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:58.603 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:44:58.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:44:58.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:58.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:44:58.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:44:59.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:44:59.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:44:59.970 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:44:59.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:44:59.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:44:59.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:00.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:01.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:01.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:01.304 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:45:01.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:45:01.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:01.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:01.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:02.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:02.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:02.870 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:45:02.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:45:02.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:02.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:03.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:04.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:04.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:04.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:45:04.205 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:45:04.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:04.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:04.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:05.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:05.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:05.534 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:45:05.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:45:05.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:05.535 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:05.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:06.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:06.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:06.870 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:45:06.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:45:06.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:06.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:07.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:08.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:08.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:08.693 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:45:08.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:45:08.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:08.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:09.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:10.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:10.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:10.428 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:45:10.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:45:10.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:10.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:11.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:11.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:11.906 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:45:11.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:45:11.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:11.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:12.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:13.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:13.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:13.795 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:45:13.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:45:13.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:13.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:14.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:15.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:15.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:15.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:45:15.570 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:45:15.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:15.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:16.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:17.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:45:17.225 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:45:17.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:17.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:17.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:18.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:18.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:18.635 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:45:18.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:45:18.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:18.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:45:18.990 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1063072303,v1:127.0.0.1:6819/1063072303] [v2:127.0.0.1:6820/1063072303,v1:127.0.0.1:6821/1063072303] exists,up a34ae5bb-24d2-4d50-9051-39b713d422b1 2024-09-17T18:45:18.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:45:18.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:45:18.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:45:18.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-17T18:45:18.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T18:45:19.309 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T18:45:19.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T18:45:19.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T18:45:19.760 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T18:45:19.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:45:20.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T18:45:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-17T18:45:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:45:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:45:23.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:45:23.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:45:23.454 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:45:23.454 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:45:23.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:45:23.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:45:23.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:45:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:45:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:45:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:45:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:45:23.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:45:23.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:45:23.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:45:23.922 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:45:23.922 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:45:23.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:45:23.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:23.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:45:24.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-17T18:45:24.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-17T18:45:24.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-17T18:45:24.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:24.074 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:45:24.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=73014444046 2024-09-17T18:45:24.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 73014444046 2024-09-17T18:45:24.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-73014444046' 2024-09-17T18:45:24.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:24.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:45:24.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855366 2024-09-17T18:45:24.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855366 2024-09-17T18:45:24.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-73014444046 2-150323855366' 2024-09-17T18:45:24.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:24.359 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-17T18:45:24.359 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:24.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:45:24.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-17T18:45:24.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:24.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-17T18:45:24.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-17T18:45:24.362 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836501 2024-09-17T18:45:24.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:24.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836501 2024-09-17T18:45:24.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:45:25.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:45:25.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:26.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-17T18:45:26.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:26.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-73014444046 2024-09-17T18:45:26.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:26.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:45:26.111 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-73014444046 2024-09-17T18:45:26.111 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:26.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=73014444046 2024-09-17T18:45:26.113 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 73014444046 2024-09-17T18:45:26.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 73014444046' 2024-09-17T18:45:26.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:45:26.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 73014444046 -lt 73014444046 2024-09-17T18:45:26.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:26.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855366 2024-09-17T18:45:26.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:45:26.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855366 2024-09-17T18:45:26.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:26.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855366 2024-09-17T18:45:26.430 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 150323855366 2024-09-17T18:45:26.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855366' 2024-09-17T18:45:26.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:45:26.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855366 -lt 150323855366 2024-09-17T18:45:26.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:45:26.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:26.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:27.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:45:27.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:45:27.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:45:27.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:45:27.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:45:27.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:45:27.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:45:27.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:45:27.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:45:27.530 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:45:27.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:27.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T18:45:27.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-17T18:45:28.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T18:45:28.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T18:45:29.148 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T18:45:29.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:45:30.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-17T18:45:30.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T18:45:30.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:45:30.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:45:30.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:45:30.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:45:30.172 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:45:30.172 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:45:30.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:45:30.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:45:30.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:45:30.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:45:30.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:45:30.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:45:30.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:45:30.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:45:30.299 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:45:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:45:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:45:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:45:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:45:30.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:30.642 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:45:30.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-17T18:45:30.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-17T18:45:30.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-17T18:45:30.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:30.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:45:30.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=73014444049 2024-09-17T18:45:30.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 73014444049 2024-09-17T18:45:30.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-73014444049' 2024-09-17T18:45:30.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:30.971 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:45:31.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855368 2024-09-17T18:45:31.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855368 2024-09-17T18:45:31.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-73014444049 2-150323855368' 2024-09-17T18:45:31.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:31.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-17T18:45:31.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:31.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:45:31.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-17T18:45:31.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:31.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-17T18:45:31.118 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836503 2024-09-17T18:45:31.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-17T18:45:31.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:31.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-17T18:45:31.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:31.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-73014444049 2024-09-17T18:45:31.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:31.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:45:31.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-73014444049 2024-09-17T18:45:31.446 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:31.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=73014444049 2024-09-17T18:45:31.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 73014444049' 2024-09-17T18:45:31.447 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 73014444049 2024-09-17T18:45:31.448 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:45:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 73014444048 -lt 73014444049 2024-09-17T18:45:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:45:32.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:45:32.762 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:45:33.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 73014444049 -lt 73014444049 2024-09-17T18:45:33.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:33.094 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855368 2024-09-17T18:45:33.094 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:33.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:45:33.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855368 2024-09-17T18:45:33.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:33.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855368 2024-09-17T18:45:33.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855368' 2024-09-17T18:45:33.099 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 150323855368 2024-09-17T18:45:33.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:45:33.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855368 -lt 150323855368 2024-09-17T18:45:33.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:45:33.446 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:33.446 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:45:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:45:33.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:45:33.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:45:33.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:45:33.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:45:33.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:45:33.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:45:34.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:45:34.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:45:34.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:34.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-17T18:45:34.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:45:34.831 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-17T18:45:34.831 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-17T18:45:34.831 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:45:34.832 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-17T18:45:34.832 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T18:45:34.832 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:45:34.832 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:45:34.832 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:45:35.193 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-17T18:45:35.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:45:35.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:45:35.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:45:35.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:45:35.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:45:35.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:45:35.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:45:35.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:45:35.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:45:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:45:38.010 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T18:45:38.010 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-17T18:45:38.010 INFO:tasks.workunit.client.0.smithi105.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:45:38.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:45:38.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:45:38.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:45:38.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:45:38.548 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:45:38.548 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:45:38.548 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:45:38.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:45:38.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:45:38.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:45:38.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:45:38.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:45:38.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:45:38.551 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:45:38.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:45:38.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:45:38.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:45:38.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:45:38.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:45:38.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:45:38.588 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:45:38.587+0000 7f12fe4d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:45:38.590 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:45:38.587+0000 7f12fe4d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:45:38.592 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:45:38.591+0000 7f12fe4d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:45:38.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:45:38.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:45:38.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:45:38.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:45:38.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:45:38.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:38.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:45:38.873 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:45:38.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:38.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:45:39.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:39.907 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:45:39.903+0000 7f12fe4d27c0 -1 Falling back to public interface 2024-09-17T18:45:40.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:40.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:40.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:45:40.233 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:45:40.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:40.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:45:40.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:41.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:41.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:41.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:45:41.567 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:45:41.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:41.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:45:41.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:45:42.706 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:45:42.702+0000 7f12fe4d27c0 -1 osd.1 57 log_to_monitors true 2024-09-17T18:45:42.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:45:42.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:45:42.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:45:42.904 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:45:42.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:45:42.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:45:43.377 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 64 up_thru 37 down_at 58 last_clean_interval [17,57) [v2:127.0.0.1:6810/867308217,v1:127.0.0.1:6811/867308217] [v2:127.0.0.1:6812/867308217,v1:127.0.0.1:6813/867308217] exists,up 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 2024-09-17T18:45:43.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:45:43.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:45:43.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:45:43.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:45:43.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:45:43.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:45:43.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:45:43.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:45:43.380 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:45:43.380 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:45:43.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:45:43.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:45:43.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:45:43.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:45:43.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:45:43.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:45:43.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:45:43.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:45:43.485 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:45:43.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:45:43.888 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:45:43.888 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:45:43.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:45:43.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:43.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:45:44.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-17T18:45:44.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-17T18:45:44.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-17T18:45:44.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:44.034 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:45:44.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906946 2024-09-17T18:45:44.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906946 2024-09-17T18:45:44.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-274877906946' 2024-09-17T18:45:44.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:44.303 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:45:44.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855372 2024-09-17T18:45:44.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855372 2024-09-17T18:45:44.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-274877906946 2-150323855372' 2024-09-17T18:45:44.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:44.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-17T18:45:44.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:44.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:45:44.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-17T18:45:44.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:44.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-17T18:45:44.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-17T18:45:44.472 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836507 2024-09-17T18:45:44.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:45.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-17T18:45:45.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:45:46.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:45:46.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:46.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-17T18:45:46.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:46.342 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-274877906946 2024-09-17T18:45:46.342 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:46.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:45:46.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-274877906946 2024-09-17T18:45:46.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:46.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906946 2024-09-17T18:45:46.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 274877906946' 2024-09-17T18:45:46.345 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 274877906946 2024-09-17T18:45:46.346 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:45:46.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906946 -lt 274877906946 2024-09-17T18:45:46.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:46.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855372 2024-09-17T18:45:46.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:46.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:45:46.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855372 2024-09-17T18:45:46.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:46.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855372 2024-09-17T18:45:46.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2024-09-17T18:45:46.822 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 150323855372 2024-09-17T18:45:46.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:45:47.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855372 -lt 150323855372 2024-09-17T18:45:47.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:45:47.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:47.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:47.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:45:47.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:45:47.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:45:47.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:45:47.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:45:47.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:45:47.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:45:47.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:45:47.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:45:47.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:45:47.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:47.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:48.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-17T18:45:48.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' -1 2024-09-17T18:45:48.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:45:48.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-17T18:45:48.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:45:48.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:45:48.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:45:48.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:45:48.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:45:48.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:45:48.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:45:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:45:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:45:48.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:45:48.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-17T18:45:48.685 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:45:48.685 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:48.686 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:49.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 5 2024-09-17T18:45:49.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-17T18:45:49.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:45:49.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:45:49.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:45:49.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:45:49.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:45:49.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:45:49.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:45:49.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:45:49.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:45:49.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T18:45:49.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:45:49.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:45:49.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:45:49.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:45:49.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:45:49.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:45:49.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:45:49.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:45:49.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:45:49.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:45:49.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:45:49.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:45:49.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:49.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:45:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:45:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:45:50.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-17T18:45:50.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:45:50.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:45:50.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:45:50.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:45:50.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-17T18:45:50.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-17T18:45:50.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:45:50.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:45:50.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:45:50.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:45:51.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-17T18:45:29.031691+0000 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:45:29.031691+0000 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:45:51.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:45:51.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:45:29.031691+0000 '>' 2024-09-17T18:45:29.031691+0000 2024-09-17T18:45:51.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:45:52.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:45:52.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:45:52.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:45:52.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:45:52.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:45:52.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:45:52.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:45:52.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:45:29.031691+0000 '>' 2024-09-17T18:45:29.031691+0000 2024-09-17T18:45:52.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:45:53.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:45:53.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:45:53.849 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:45:53.849 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:45:53.849 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:45:53.849 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:45:53.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:45:54.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:45:29.031691+0000 '>' 2024-09-17T18:45:29.031691+0000 2024-09-17T18:45:54.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:45:55.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:45:55.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:45:55.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:45:55.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:45:55.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:45:55.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:45:55.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:45:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:45:50.584945+0000 '>' 2024-09-17T18:45:29.031691+0000 2024-09-17T18:45:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:45:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-17T18:45:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:45:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:45:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:45:55.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:45:55.517 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:45:55.517 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:45:55.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:45:55.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:45:55.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:45:55.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:45:55.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:45:55.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:45:55.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:45:55.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:45:55.629 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:45:55.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:45:55.960 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:45:55.960 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:45:55.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:45:55.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:55.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:45:56.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836511 2024-09-17T18:45:56.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836511 2024-09-17T18:45:56.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511' 2024-09-17T18:45:56.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:56.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:45:56.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906949 2024-09-17T18:45:56.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906949 2024-09-17T18:45:56.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-274877906949' 2024-09-17T18:45:56.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:45:56.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:45:56.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855375 2024-09-17T18:45:56.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855375 2024-09-17T18:45:56.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836511 1-274877906949 2-150323855375' 2024-09-17T18:45:56.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:56.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836511 2024-09-17T18:45:56.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:56.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:45:56.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836511 2024-09-17T18:45:56.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:56.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836511 2024-09-17T18:45:56.418 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836511 2024-09-17T18:45:56.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-17T18:45:56.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:56.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-17T18:45:56.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:45:57.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:45:57.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:45:58.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-17T18:45:58.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:58.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-274877906949 2024-09-17T18:45:58.106 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:58.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:45:58.108 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-274877906949 2024-09-17T18:45:58.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:58.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906949 2024-09-17T18:45:58.110 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 274877906949 2024-09-17T18:45:58.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 274877906949' 2024-09-17T18:45:58.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:45:58.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906949 -lt 274877906949 2024-09-17T18:45:58.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:45:58.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855375 2024-09-17T18:45:58.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:45:58.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:45:58.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855375 2024-09-17T18:45:58.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:45:58.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855375 2024-09-17T18:45:58.455 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 150323855375 2024-09-17T18:45:58.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855375' 2024-09-17T18:45:58.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:45:58.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855375 -lt 150323855375 2024-09-17T18:45:58.776 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:45:58.776 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:58.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:59.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:45:59.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:45:59.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:45:59.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:45:59.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:45:59.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:45:59.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:45:59.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:45:59.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:45:59.545 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:45:59.545 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:45:59.545 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:45:59.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:45:59.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:45:59.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:45:59.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-17T18:45:59.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-17T18:45:59.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:45:59.914 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-17T18:45:59.914 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T18:45:59.914 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:45:59.914 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:45:59.914 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:46:00.242 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-17T18:46:00.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:46:00.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-17T18:46:00.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:46:00.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:46:00.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:46:00.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:46:00.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:46:00.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:46:01.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:46:01.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:46:01.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:46:01.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:46:01.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:46:01.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:46:01.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:46:01.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:46:02.361 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T18:46:02.361 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-17T18:46:02.361 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:46:02.361 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T18:46:02.361 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:46:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:02.657 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:02.658 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:02.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:46:02.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:46:02.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:46:02.660 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:46:02.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:46:02.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:46:02.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:46:02.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:46:02.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:46:02.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:46:02.697 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:02.694+0000 7f78196cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:02.733 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:02.730+0000 7f78196cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:02.735 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:02.730+0000 7f78196cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:02.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:03.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:03.823 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:03.818+0000 7f78196cd7c0 -1 Falling back to public interface 2024-09-17T18:46:04.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:04.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:04.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:46:04.405 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:46:04.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:04.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:04.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:05.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:05.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:05.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:46:05.727 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:46:05.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:05.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:06.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:07.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:07.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:07.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:46:07.061 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:46:07.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:07.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:07.182 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:07.178+0000 7f78196cd7c0 -1 osd.1 74 log_to_monitors true 2024-09-17T18:46:07.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:08.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:08.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:08.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:46:08.480 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:46:08.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:08.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:08.817 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 81 up_thru 81 down_at 75 last_clean_interval [64,74) [v2:127.0.0.1:6810/3739470601,v1:127.0.0.1:6811/3739470601] [v2:127.0.0.1:6812/3739470601,v1:127.0.0.1:6813/3739470601] exists,up 2fdb2264-6bba-4bb2-b2d8-af8c5451aca4 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:46:08.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:46:08.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:46:08.819 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:46:08.819 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:46:08.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:46:08.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:46:08.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:46:08.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:46:08.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:46:08.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:46:08.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:46:08.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:46:08.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:46:09.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:46:09.273 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:46:09.273 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:46:09.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:46:09.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:46:09.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:46:09.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836514 2024-09-17T18:46:09.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836514 2024-09-17T18:46:09.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514' 2024-09-17T18:46:09.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:46:09.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:46:09.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=347892350978 2024-09-17T18:46:09.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 347892350978 2024-09-17T18:46:09.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-347892350978' 2024-09-17T18:46:09.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:46:09.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:46:09.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855379 2024-09-17T18:46:09.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855379 2024-09-17T18:46:09.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836514 1-347892350978 2-150323855379' 2024-09-17T18:46:09.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:46:09.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836514 2024-09-17T18:46:09.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:46:09.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:46:09.765 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836514 2024-09-17T18:46:09.765 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:46:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836514 2024-09-17T18:46:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2024-09-17T18:46:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836514 2024-09-17T18:46:09.766 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:46:10.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836514 2024-09-17T18:46:10.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:46:11.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:46:11.094 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:46:11.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836514 2024-09-17T18:46:11.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:46:11.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-347892350978 2024-09-17T18:46:11.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:46:11.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:46:11.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-347892350978 2024-09-17T18:46:11.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:46:11.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=347892350978 2024-09-17T18:46:11.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 347892350978' 2024-09-17T18:46:11.426 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 347892350978 2024-09-17T18:46:11.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:46:11.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 347892350978 -lt 347892350978 2024-09-17T18:46:11.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:46:11.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-150323855379 2024-09-17T18:46:11.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:46:11.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:46:11.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-150323855379 2024-09-17T18:46:11.758 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:46:11.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855379 2024-09-17T18:46:11.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 150323855379' 2024-09-17T18:46:11.759 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 150323855379 2024-09-17T18:46:11.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:46:12.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855379 -lt 150323855379 2024-09-17T18:46:12.081 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:46:12.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:46:12.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:46:12.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:46:12.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:46:12.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:46:12.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:46:12.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:46:12.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:46:12.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:46:12.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:46:12.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:46:12.831 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:46:12.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:46:12.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:46:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:46:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:46:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:46:13.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:46:13.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:46:13.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:46:13.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:46:13.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:46:13.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:46:13.351 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:46:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:46:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:46:13.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:46:13.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:46:13.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:46:13.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:46:13.353 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:46:13.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:46:13.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:46:13.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:46:13.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:46:13.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:13.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:13.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:46:13.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:46:13.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:46:13.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:46:13.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:46:13.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:46:13.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:46:13.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:46:13.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:46:13.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:46:13.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:46:13.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:46:13.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:46:13.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:46:13.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:46:13.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:46:13.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:46:13.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:46:13.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:13.384 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:13.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:46:13.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:46:13.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:46:13.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:46:13.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:46:13.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:13.386 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:13.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-09-17T18:46:13.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:46:13.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:46:13.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:46:13.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:46:13.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:46:13.874 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:46:13.875 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:13.875 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:13.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:13.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:46:14.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:46:14.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:46:14.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:46:14.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:46:14.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:46:14.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:46:14.088 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:46:14.088 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:46:14.088 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:46:14.089 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:46:14.089 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:14.089 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:14.089 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:46:14.089 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:46:14.089 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:46:14.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:46:14.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:46:14.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:46:14.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:14.205 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:46:14.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:46:14.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:46:14.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-09-17T18:46:14.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:46:14.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:46:14.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:46:14.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:46:14.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:46:14.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:46:15.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:46:15.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:46:15.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:46:15.186 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:46:15.186 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:15.186 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:15.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:15.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:46:15.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:46:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:46:15.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-09-17T18:46:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-17T18:46:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:46:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:46:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:46:15.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:46:15.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:46:15.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=96c6acf3-bf8b-4ada-bb2e-ddd2e4fa8507 2024-09-17T18:46:15.228 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 96c6acf3-bf8b-4ada-bb2e-ddd2e4fa8507 2024-09-17T18:46:15.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 96c6acf3-bf8b-4ada-bb2e-ddd2e4fa8507' 2024-09-17T18:46:15.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:46:15.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB3zulmSPSODhAAIOj+Hcc8g4WMWw7E1Qr8yg== 2024-09-17T18:46:15.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB3zulmSPSODhAAIOj+Hcc8g4WMWw7E1Qr8yg=="}' 2024-09-17T18:46:15.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 96c6acf3-bf8b-4ada-bb2e-ddd2e4fa8507 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:46:15.505 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:46:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:46:15.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB3zulmSPSODhAAIOj+Hcc8g4WMWw7E1Qr8yg== --osd-uuid 96c6acf3-bf8b-4ada-bb2e-ddd2e4fa8507 2024-09-17T18:46:15.559 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:15.557+0000 7f834a4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:15.561 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:15.561+0000 7f834a4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:15.563 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:15.561+0000 7f834a4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:15.563 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:15.561+0000 7f834a4c37c0 -1 bdev(0x5571c348fc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:46:15.563 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:15.561+0000 7f834a4c37c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:46:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:46:21.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:46:21.847 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:46:21.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:46:21.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:46:22.261 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:46:22.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:46:22.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:46:22.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:46:22.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:46:22.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:46:22.298 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:22.297+0000 7f3a126c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:22.326 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:22.325+0000 7f3a126c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:22.328 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:22.329+0000 7f3a126c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:22.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:46:22.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:46:22.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:22.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:23.819 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:23.817+0000 7f3a126c97c0 -1 Falling back to public interface 2024-09-17T18:46:23.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:23.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:23.962 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:46:23.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:46:23.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:23.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:24.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:25.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:25.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:25.284 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:46:25.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:46:25.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:25.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:25.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:26.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:26.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:26.597 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:46:26.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:46:26.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:26.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:26.845 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:26.845+0000 7f3a126c97c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:46:26.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:27.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:27.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:27.949 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:46:27.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:46:27.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:27.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:28.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:29.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:29.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:29.281 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:46:29.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:46:29.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:29.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:29.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:30.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:30.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:30.615 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:46:30.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:46:30.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:30.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:30.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:31.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:31.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:31.945 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:46:31.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:46:31.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:31.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:32.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:33.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:33.273 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:46:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:46:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:33.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:34.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:34.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:34.633 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:46:34.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:46:34.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:34.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:34.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:35.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:35.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:35.973 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:46:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:46:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:35.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:36.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:37.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:37.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:37.314 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:46:37.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:46:37.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:37.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:37.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:38.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:38.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:38.672 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:46:38.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:46:38.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:38.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:39.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:40.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:40.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:40.057 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:46:40.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:46:40.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:40.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:40.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:41.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:41.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:41.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:46:41.932 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:46:41.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:41.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:42.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:43.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:43.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:43.466 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:46:43.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:46:43.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:43.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:44.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:45.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:45.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:45.056 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:46:45.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:46:45.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:45.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:45.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:46.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:46.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:46.824 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:46:46.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:46:46.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:46.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:47.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:48.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:48.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:48.183 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:46:48.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:46:48.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:48.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:49.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:49.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:49.961 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:46:49.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:46:49.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:46:50.281 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3624721814,v1:127.0.0.1:6803/3624721814] [v2:127.0.0.1:6804/3624721814,v1:127.0.0.1:6805/3624721814] exists,up 96c6acf3-bf8b-4ada-bb2e-ddd2e4fa8507 2024-09-17T18:46:50.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:46:50.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:46:50.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:46:50.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:46:50.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:46:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:46:50.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:46:50.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:46:50.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dd2fa93b-521b-4f8a-bf84-e10fed189695 2024-09-17T18:46:50.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 dd2fa93b-521b-4f8a-bf84-e10fed189695' 2024-09-17T18:46:50.289 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 dd2fa93b-521b-4f8a-bf84-e10fed189695 2024-09-17T18:46:50.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:46:50.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCazulmzdoiEhAAvOn0D4Dkbv8YJD4B4xVzYA== 2024-09-17T18:46:50.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCazulmzdoiEhAAvOn0D4Dkbv8YJD4B4xVzYA=="}' 2024-09-17T18:46:50.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dd2fa93b-521b-4f8a-bf84-e10fed189695 -i td/osd-scrub-repair/1/new.json 2024-09-17T18:46:50.681 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:46:50.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:46:50.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCazulmzdoiEhAAvOn0D4Dkbv8YJD4B4xVzYA== --osd-uuid dd2fa93b-521b-4f8a-bf84-e10fed189695 2024-09-17T18:46:50.741 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:50.741+0000 7f36203147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:50.765 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:50.765+0000 7f36203147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:50.767 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:50.765+0000 7f36203147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:50.768 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:50.765+0000 7f36203147c0 -1 bdev(0x55b3b0dde800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:46:50.768 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:50.765+0000 7f36203147c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:46:57.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:46:57.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:46:57.397 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:46:57.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:46:57.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:46:57.812 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:46:57.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:46:57.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:46:57.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:46:57.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:46:57.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:46:57.848 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:57.848+0000 7fde3861d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:57.880 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:57.880+0000 7fde3861d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:57.882 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:57.880+0000 7fde3861d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:46:58.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:46:58.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:46:58.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:46:58.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:46:58.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:46:58.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:58.218 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:46:58.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:46:58.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:58.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:58.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:46:59.159 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:46:59.156+0000 7fde3861d7c0 -1 Falling back to public interface 2024-09-17T18:46:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:46:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:46:59.558 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:46:59.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:46:59.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:46:59.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:46:59.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:00.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:00.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:00.890 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:47:00.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:47:00.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:00.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:01.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:02.003 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:02.000+0000 7fde3861d7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:47:02.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:02.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:02.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:47:02.225 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:47:02.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:02.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:02.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:03.611 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:47:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:47:03.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:03.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:03.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:04.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:04.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:04.928 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:47:04.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:47:04.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:04.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:05.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:06.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:06.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:06.301 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:47:06.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:47:06.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:06.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:06.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:07.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:07.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:07.638 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:47:07.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:47:07.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:07.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:07.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:08.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:08.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:08.972 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:47:08.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:47:08.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:08.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:09.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:10.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:10.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:10.309 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:47:10.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:47:10.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:10.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:10.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:11.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:11.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:11.820 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:47:11.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:47:11.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:11.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:12.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:13.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:13.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:13.251 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:47:13.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:47:13.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:13.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:13.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:14.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:14.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:14.589 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:47:14.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:47:14.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:14.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:15.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:16.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:16.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:16.209 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:47:16.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:47:16.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:16.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:17.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:18.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:18.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:18.230 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:47:18.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:47:18.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:18.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:18.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:19.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:19.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:19.865 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:47:19.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:47:19.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:19.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:20.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:21.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:21.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:21.775 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:47:21.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:47:21.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:21.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:22.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:23.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:23.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:23.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:47:23.289 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:47:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:24.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:25.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:25.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:25.110 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:47:25.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:47:25.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:25.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:25.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:26.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:26.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:26.455 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:47:26.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:47:26.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:26.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:47:26.785 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1994616661,v1:127.0.0.1:6811/1994616661] [v2:127.0.0.1:6812/1994616661,v1:127.0.0.1:6813/1994616661] exists,up dd2fa93b-521b-4f8a-bf84-e10fed189695 2024-09-17T18:47:26.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:47:26.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:47:26.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:47:26.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:47:26.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:47:26.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:47:26.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:47:26.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:47:26.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:47:26.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:47:26.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=142aff6d-6f01-4ff7-82a2-6d183589d047 2024-09-17T18:47:26.793 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 142aff6d-6f01-4ff7-82a2-6d183589d047 2024-09-17T18:47:26.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 142aff6d-6f01-4ff7-82a2-6d183589d047' 2024-09-17T18:47:26.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:47:26.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC+zulmuD52MBAApuIp5xPhxClQCC68Xro9FQ== 2024-09-17T18:47:26.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC+zulmuD52MBAApuIp5xPhxClQCC68Xro9FQ=="}' 2024-09-17T18:47:26.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 142aff6d-6f01-4ff7-82a2-6d183589d047 -i td/osd-scrub-repair/2/new.json 2024-09-17T18:47:27.191 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:47:27.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:47:27.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQC+zulmuD52MBAApuIp5xPhxClQCC68Xro9FQ== --osd-uuid 142aff6d-6f01-4ff7-82a2-6d183589d047 2024-09-17T18:47:27.252 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:27.248+0000 7f761e4137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:47:27.284 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:27.284+0000 7f761e4137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:47:27.286 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:27.284+0000 7f761e4137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:47:27.286 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:27.284+0000 7f761e4137c0 -1 bdev(0x55fecf90fc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:47:27.286 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:27.284+0000 7f761e4137c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:47:34.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:47:34.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:47:34.270 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:47:34.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:47:34.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:47:34.680 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:47:34.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:47:34.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:47:34.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:47:34.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:47:34.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:47:34.717 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:34.715+0000 7f056da067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:47:34.743 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:34.739+0000 7f056da067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:47:34.745 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:34.743+0000 7f056da067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:47:35.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:47:35.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:47:35.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:47:35.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:47:35.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:47:35.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:35.009 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:47:35.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:47:35.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:35.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:35.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:35.816 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:35.811+0000 7f056da067c0 -1 Falling back to public interface 2024-09-17T18:47:36.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:36.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:36.346 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:47:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:47:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:36.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:37.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:37.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:37.664 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:47:37.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:47:37.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:37.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:38.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:38.681 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:47:38.679+0000 7f056da067c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:47:39.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:39.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:39.014 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:47:39.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:47:39.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:39.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:39.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:40.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:40.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:40.448 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:47:40.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:47:40.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:40.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:40.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:41.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:41.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:41.818 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:47:41.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:47:41.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:41.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:42.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:43.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:43.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:43.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:47:43.230 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:47:43.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:43.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:43.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:44.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:44.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:44.556 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:47:44.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:47:44.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:44.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:44.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:45.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:45.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:45.889 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:47:45.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:47:45.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:45.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:46.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:47.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:47.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:47.219 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:47:47.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:47:47.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:47.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:47.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:48.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:48.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:48.709 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:47:48.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:47:48.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:48.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:49.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:50.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:50.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:50.109 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:47:50.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:47:50.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:50.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:50.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:51.587 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:47:51.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:47:51.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:51.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:51.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:52.941 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:47:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:47:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:53.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:54.756 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:47:54.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:47:54.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:54.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:55.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:56.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:56.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:56.492 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:47:56.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:47:56.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:56.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:57.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:58.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:58.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:58.285 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:47:58.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:47:58.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:58.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:47:58.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:47:59.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:47:59.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:47:59.996 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:47:59.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:47:59.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:47:59.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:48:00.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:01.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:01.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:01.705 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:48:01.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:48:01.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:01.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:48:02.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:03.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:03.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:03.310 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:48:03.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:48:03.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:03.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:48:03.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:04.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:04.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:04.677 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:48:04.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:48:04.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:04.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:48:04.993 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1467455194,v1:127.0.0.1:6819/1467455194] [v2:127.0.0.1:6820/1467455194,v1:127.0.0.1:6821/1467455194] exists,up 142aff6d-6f01-4ff7-82a2-6d183589d047 2024-09-17T18:48:04.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:48:04.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:48:04.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:48:04.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-09-17T18:48:04.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T18:48:05.349 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T18:48:05.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T18:48:05.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T18:48:05.811 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T18:48:05.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:48:06.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T18:48:09.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-09-17T18:48:09.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:48:09.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:48:09.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:48:09.356 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:48:09.357 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:48:09.357 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:48:09.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:48:09.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:48:09.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:48:09.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:48:09.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:48:09.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:48:09.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:48:09.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:48:09.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:48:09.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:48:09.838 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:48:09.838 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:48:09.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:48:09.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:09.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:48:09.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836500 2024-09-17T18:48:09.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836500 2024-09-17T18:48:09.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500' 2024-09-17T18:48:09.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:09.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:48:10.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T18:48:10.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T18:48:10.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-64424509455' 2024-09-17T18:48:10.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:10.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:48:10.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920774 2024-09-17T18:48:10.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920774 2024-09-17T18:48:10.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836500 1-64424509455 2-141733920774' 2024-09-17T18:48:10.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:10.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836500 2024-09-17T18:48:10.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:10.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:48:10.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836500 2024-09-17T18:48:10.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:10.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836500 2024-09-17T18:48:10.274 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836500 2024-09-17T18:48:10.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-17T18:48:10.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:10.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-17T18:48:10.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:48:11.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:48:11.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:11.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836500 2024-09-17T18:48:11.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:48:12.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T18:48:12.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:13.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-17T18:48:13.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:13.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-64424509455 2024-09-17T18:48:13.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:13.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:48:13.293 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-64424509455 2024-09-17T18:48:13.293 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:13.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T18:48:13.294 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 64424509455 2024-09-17T18:48:13.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 64424509455' 2024-09-17T18:48:13.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:48:13.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509455 2024-09-17T18:48:13.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:13.634 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920774 2024-09-17T18:48:13.634 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:13.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:48:13.636 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920774 2024-09-17T18:48:13.636 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:13.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920774 2024-09-17T18:48:13.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920774' 2024-09-17T18:48:13.638 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 141733920774 2024-09-17T18:48:13.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:48:13.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920774 -lt 141733920774 2024-09-17T18:48:13.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:48:13.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:13.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:14.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:48:14.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:48:14.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:48:14.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:48:14.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:48:14.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:48:14.398 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:48:14.398 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:48:14.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:48:14.729 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:48:14.729 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:14.729 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:15.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T18:48:15.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-17T18:48:15.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T18:48:15.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T18:48:16.697 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T18:48:16.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:48:17.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-17T18:48:17.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-17T18:48:18.245 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 2 allow_ec_overwrites to true 2024-09-17T18:48:18.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T18:48:18.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:48:18.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:48:18.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:48:18.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:48:18.267 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:48:18.267 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:48:18.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:48:18.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:48:18.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:48:18.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:48:18.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:48:18.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:48:18.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:48:18.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:48:18.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:48:18.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:48:18.739 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:48:18.739 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:48:18.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:48:18.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:18.740 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:48:18.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-17T18:48:18.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-17T18:48:18.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-17T18:48:18.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:18.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:48:19.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-17T18:48:19.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-17T18:48:19.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-64424509458' 2024-09-17T18:48:19.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:19.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:48:19.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920776 2024-09-17T18:48:19.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920776 2024-09-17T18:48:19.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-64424509458 2-141733920776' 2024-09-17T18:48:19.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:19.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-17T18:48:19.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:19.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:48:19.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-17T18:48:19.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:19.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-17T18:48:19.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-17T18:48:19.196 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836503 2024-09-17T18:48:19.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:19.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-17T18:48:19.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:48:20.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:48:20.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:20.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-17T18:48:20.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:20.835 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:20.835 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-64424509458 2024-09-17T18:48:20.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:48:20.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-64424509458 2024-09-17T18:48:20.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:20.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-17T18:48:20.838 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 64424509458 2024-09-17T18:48:20.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 64424509458' 2024-09-17T18:48:20.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:48:21.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-17T18:48:21.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:21.169 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920776 2024-09-17T18:48:21.169 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:21.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:48:21.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920776 2024-09-17T18:48:21.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:21.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920776 2024-09-17T18:48:21.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920776' 2024-09-17T18:48:21.173 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 141733920776 2024-09-17T18:48:21.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:48:21.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920777 -lt 141733920776 2024-09-17T18:48:21.516 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:48:21.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:21.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:21.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:48:21.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:48:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:48:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:48:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:48:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:48:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:48:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:48:22.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:48:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:48:22.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:22.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-09-17T18:48:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:48:22.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-17T18:48:22.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-17T18:48:22.812 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:48:22.812 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-17T18:48:22.812 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T18:48:22.813 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:48:22.813 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:48:22.813 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:48:23.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-17T18:48:23.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:48:23.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:48:23.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:48:23.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:48:23.496 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:48:23.496 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:48:23.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:48:23.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:48:23.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:48:23.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:48:23.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:48:23.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:48:23.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:48:23.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:48:23.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:48:23.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:48:23.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:48:26.378 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T18:48:26.379 INFO:tasks.workunit.client.0.smithi105.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:48:26.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:48:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:48:26.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:48:26.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:48:26.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:48:26.920 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:48:26.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:48:26.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:48:26.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:48:26.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:48:26.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:48:26.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:48:26.958 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:26.954+0000 7fa1ded307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:48:26.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:26.958+0000 7fa1ded307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:48:26.962 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:26.958+0000 7fa1ded307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:27.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:48:27.261 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:48:27.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:27.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:27.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:28.456 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:28.450+0000 7fa1ded307c0 -1 Falling back to public interface 2024-09-17T18:48:28.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:28.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:28.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:48:28.747 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:48:28.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:28.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:29.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:30.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:30.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:30.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:48:30.062 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:48:30.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:30.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:30.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:31.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:31.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:31.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:48:31.383 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:48:31.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:31.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:31.529 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:31.526+0000 7fa1ded307c0 -1 osd.1 57 log_to_monitors true 2024-09-17T18:48:31.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:32.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:32.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:32.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:48:32.767 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:48:32.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:32.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:33.205 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 58 last_clean_interval [15,57) [v2:127.0.0.1:6810/4044498898,v1:127.0.0.1:6811/4044498898] [v2:127.0.0.1:6812/4044498898,v1:127.0.0.1:6813/4044498898] exists,up dd2fa93b-521b-4f8a-bf84-e10fed189695 2024-09-17T18:48:33.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:48:33.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:48:33.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:48:33.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:48:33.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:48:33.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:48:33.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:48:33.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:48:33.206 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:48:33.207 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:48:33.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:48:33.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:48:33.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:48:33.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:48:33.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:48:33.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:48:33.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:48:33.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:48:33.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:48:33.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:48:33.697 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:48:33.697 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:48:33.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:48:33.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:33.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:48:33.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836507 2024-09-17T18:48:33.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836507 2024-09-17T18:48:33.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507' 2024-09-17T18:48:33.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:33.898 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:48:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874242 2024-09-17T18:48:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874242 2024-09-17T18:48:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-279172874242' 2024-09-17T18:48:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:34.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:48:34.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920780 2024-09-17T18:48:34.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920780 2024-09-17T18:48:34.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836507 1-279172874242 2-141733920780' 2024-09-17T18:48:34.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:34.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836507 2024-09-17T18:48:34.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:34.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:48:34.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836507 2024-09-17T18:48:34.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:34.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836507 2024-09-17T18:48:34.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-17T18:48:34.195 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836507 2024-09-17T18:48:34.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:34.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-17T18:48:34.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:48:35.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:48:35.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:35.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836507 2024-09-17T18:48:35.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:48:36.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T18:48:36.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:37.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-17T18:48:37.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:37.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-279172874242 2024-09-17T18:48:37.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:37.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:48:37.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-279172874242 2024-09-17T18:48:37.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:37.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874242 2024-09-17T18:48:37.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-09-17T18:48:37.266 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 279172874242 2024-09-17T18:48:37.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:48:37.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874242 -lt 279172874242 2024-09-17T18:48:37.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:37.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920780 2024-09-17T18:48:37.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:37.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:48:37.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920780 2024-09-17T18:48:37.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:37.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920780 2024-09-17T18:48:37.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920780' 2024-09-17T18:48:37.591 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 141733920780 2024-09-17T18:48:37.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:48:38.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920781 -lt 141733920780 2024-09-17T18:48:38.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:48:38.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:38.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:38.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:48:38.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:48:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:48:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:48:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:48:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:48:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:48:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:48:38.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:48:38.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:48:38.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:38.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:39.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:48:39.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T18:48:39.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:48:39.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T18:48:39.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:48:39.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:48:39.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:48:39.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:48:39.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:48:39.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:48:39.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:48:39.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:48:39.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:48:39.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:48:39.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:48:39.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:48:39.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:39.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:48:39.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:48:39.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:48:39.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:48:40.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-17T18:48:40.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-17T18:48:40.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:48:40.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:48:40.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:48:40.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:48:40.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:48:40.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:48:40.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:48:40.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:48:40.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:48:40.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:48:40.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:48:40.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:48:40.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:48:40.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:40.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:41.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:48:41.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:48:41.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-09-17T18:48:41.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-09-17T18:48:41.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:41.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:41.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:41.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:41.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:41.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:48:41.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:41.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:48:41.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:48:41.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:41.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:41.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:41.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:41.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:41.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:42.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:34.178072+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:42.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:48:43.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:48:43.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:43.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:43.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:43.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:43.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:43.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:43.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:34.178072+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:43.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:48:44.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:48:44.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:44.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:44.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:44.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:44.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:44.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:44.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:34.178072+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:44.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:48:45.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:48:45.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:45.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:45.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:45.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:45.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:45.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:46.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:34.178072+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:46.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:48:47.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:48:47.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:47.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:47.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:47.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:47.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:47.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:47.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:34.178072+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:47.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:48:48.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:48:48.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:48.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:48.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:48.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:48.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:48.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:48.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:34.178072+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:48.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:48:49.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:48:49.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:48:49.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:48:49.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:48:49.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:48:49.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:48:49.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:48:50.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:48:46.315801+0000 '>' 2024-09-17T18:48:34.178072+0000 2024-09-17T18:48:50.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:48:50.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-09-17T18:48:50.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:48:50.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:48:50.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:48:50.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:48:50.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:48:50.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:48:50.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:48:50.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:48:50.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:48:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:48:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:48:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:48:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:48:50.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:50.741 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:48:50.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-17T18:48:50.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-17T18:48:50.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-17T18:48:50.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:50.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:48:51.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=279172874246 2024-09-17T18:48:51.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 279172874246 2024-09-17T18:48:51.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-279172874246' 2024-09-17T18:48:51.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:48:51.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:48:51.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920785 2024-09-17T18:48:51.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920785 2024-09-17T18:48:51.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-279172874246 2-141733920785' 2024-09-17T18:48:51.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:51.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-17T18:48:51.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:51.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:48:51.193 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-17T18:48:51.193 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:51.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-17T18:48:51.194 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836512 2024-09-17T18:48:51.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-17T18:48:51.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:51.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836510 -lt 21474836512 2024-09-17T18:48:51.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:48:52.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:48:52.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:48:52.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-17T18:48:52.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:52.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-279172874246 2024-09-17T18:48:52.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:52.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:48:52.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-279172874246 2024-09-17T18:48:52.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:52.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=279172874246 2024-09-17T18:48:52.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 279172874246' 2024-09-17T18:48:52.915 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 279172874246 2024-09-17T18:48:52.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:48:53.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 279172874247 -lt 279172874246 2024-09-17T18:48:53.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:48:53.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920785 2024-09-17T18:48:53.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:48:53.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:48:53.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920785 2024-09-17T18:48:53.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:48:53.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920785 2024-09-17T18:48:53.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920785' 2024-09-17T18:48:53.224 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 141733920785 2024-09-17T18:48:53.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:48:53.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920785 -lt 141733920785 2024-09-17T18:48:53.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:48:53.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:53.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:48:53.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:48:53.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:48:53.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:48:53.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:48:53.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:48:53.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:48:53.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:48:54.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:48:54.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:48:54.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:48:54.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:48:54.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:48:54.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:48:54.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:48:54.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-09-17T18:48:54.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2024-09-17T18:48:54.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T18:48:54.624 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2024-09-17T18:48:54.624 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T18:48:54.624 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:48:54.624 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:48:54.624 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:48:54.965 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2024-09-17T18:48:54.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-17T18:48:54.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:48:55.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:48:55.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:48:55.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:48:55.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:48:55.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:48:55.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:48:55.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:48:55.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:48:55.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:48:55.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:48:55.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:48:55.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:48:56.130 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T18:48:56.130 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-09-17T18:48:56.130 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:48:56.130 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T18:48:56.130 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:48:56.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:48:56.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:48:56.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:48:56.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:48:56.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:48:56.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:48:56.431 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:48:56.431 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:48:56.431 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-09-17T18:48:56.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:48:56.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:48:56.433 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:48:56.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-09-17T18:48:56.433 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:48:56.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:48:56.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:48:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:48:56.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:48:56.477 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:56.477+0000 7f8a7b5187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:48:56.480 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:56.477+0000 7f8a7b5187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:48:56.482 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:56.481+0000 7f8a7b5187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:48:56.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:48:56.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:48:56.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:48:56.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:48:56.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:48:56.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:56.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:48:56.792 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:48:56.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:56.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:57.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:57.528 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:48:57.525+0000 7f8a7b5187c0 -1 Falling back to public interface 2024-09-17T18:48:58.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:58.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:58.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:48:58.135 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:48:58.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:58.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:58.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:48:59.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:48:59.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:48:59.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:48:59.477 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:48:59.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:48:59.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:48:59.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:00.810 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:00.809+0000 7f8a7b5187c0 -1 osd.1 78 log_to_monitors true 2024-09-17T18:49:00.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:00.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:00.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:49:00.811 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:49:00.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:00.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:01.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:02.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:02.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:02.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:49:02.249 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:49:02.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:02.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:02.662 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 85 up_thru 85 down_at 80 last_clean_interval [65,78) [v2:127.0.0.1:6810/2675394454,v1:127.0.0.1:6811/2675394454] [v2:127.0.0.1:6812/2675394454,v1:127.0.0.1:6813/2675394454] exists,up dd2fa93b-521b-4f8a-bf84-e10fed189695 2024-09-17T18:49:02.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:49:02.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:49:02.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:49:02.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:49:02.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:49:02.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:49:02.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:49:02.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:49:02.664 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:49:02.664 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:49:02.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:49:02.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:49:02.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:49:02.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:49:02.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:49:02.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:49:02.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:49:02.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:49:02.768 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:49:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:49:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:49:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T18:49:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:49:03.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:49:03.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:49:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836515 2024-09-17T18:49:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836515 2024-09-17T18:49:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515' 2024-09-17T18:49:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:49:03.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:49:03.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220162 2024-09-17T18:49:03.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220162 2024-09-17T18:49:03.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-365072220162' 2024-09-17T18:49:03.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:49:03.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:49:03.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=141733920788 2024-09-17T18:49:03.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 141733920788 2024-09-17T18:49:03.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-365072220162 2-141733920788' 2024-09-17T18:49:03.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:49:03.733 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836515 2024-09-17T18:49:03.734 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:49:03.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:49:03.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836515 2024-09-17T18:49:03.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:49:03.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836515 2024-09-17T18:49:03.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-17T18:49:03.736 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836515 2024-09-17T18:49:03.736 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:49:04.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-17T18:49:04.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:49:04.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-365072220162 2024-09-17T18:49:04.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:49:04.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:49:04.070 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-365072220162 2024-09-17T18:49:04.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:49:04.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220162 2024-09-17T18:49:04.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 365072220162' 2024-09-17T18:49:04.073 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 365072220162 2024-09-17T18:49:04.073 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:49:04.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220162 -lt 365072220162 2024-09-17T18:49:04.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:49:04.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-141733920788 2024-09-17T18:49:04.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:49:04.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:49:04.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-141733920788 2024-09-17T18:49:04.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:49:04.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=141733920788 2024-09-17T18:49:04.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 141733920788' 2024-09-17T18:49:04.436 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 141733920788 2024-09-17T18:49:04.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:49:04.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 141733920788 -lt 141733920788 2024-09-17T18:49:04.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:49:04.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:49:04.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:49:05.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:49:05.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:49:05.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:49:05.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:49:05.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:49:05.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:49:05.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:49:05.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:49:05.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:49:05.534 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:49:05.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:49:05.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:49:05.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:49:05.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:49:05.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:49:05.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:49:06.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:49:06.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:49:06.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:49:06.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:49:06.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:49:06.004 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:49:06.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:49:06.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:49:06.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:49:06.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:49:06.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:49:06.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:49:06.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:49:06.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:49:06.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:49:06.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:49:06.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:49:06.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:49:06.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:49:06.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:49:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:49:06.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:49:06.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:49:06.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:49:06.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:49:06.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:06.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:06.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:49:06.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:49:06.140 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:49:06.140 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:49:06.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:49:06.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:49:06.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:49:06.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:49:06.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:49:06.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:49:06.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:49:06.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:49:06.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:49:06.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:49:06.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:49:06.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:49:06.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:49:06.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:49:06.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:49:06.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:49:06.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:49:06.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:49:06.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:06.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:06.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:49:06.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:49:06.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:49:06.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:49:06.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:49:06.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:06.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:06.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-09-17T18:49:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:49:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:49:06.619 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:49:06.619 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:49:06.619 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:49:06.620 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:49:06.620 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:06.620 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:06.620 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:49:06.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:49:06.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:49:06.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:49:06.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:49:06.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:49:06.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:49:06.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:49:06.825 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:49:06.825 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:49:06.825 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:49:06.826 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:49:06.826 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:06.826 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:06.826 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:49:06.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:49:06.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:49:06.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:49:06.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:49:06.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:49:06.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:49:06.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:49:06.939 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:49:06.939 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:49:06.939 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:49:06.939 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:49:06.940 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:06.940 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:06.940 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:49:06.940 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:49:06.940 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:49:07.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-17T18:49:07.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:49:07.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:49:07.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:49:07.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:49:07.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:49:07.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:49:07.921 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:49:07.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:49:07.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:49:07.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:49:07.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:49:07.963 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:49:07.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:49:07.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:49:07.965 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:49:07.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e856d44-dffd-48e9-94ec-a725f24fee80 2024-09-17T18:49:07.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9e856d44-dffd-48e9-94ec-a725f24fee80' 2024-09-17T18:49:07.965 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 9e856d44-dffd-48e9-94ec-a725f24fee80 2024-09-17T18:49:07.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:49:07.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAjz+lmNLFwOhAAlfrgRKXLKIHA4TIVXF4H4Q== 2024-09-17T18:49:07.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAjz+lmNLFwOhAAlfrgRKXLKIHA4TIVXF4H4Q=="}' 2024-09-17T18:49:07.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e856d44-dffd-48e9-94ec-a725f24fee80 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:49:08.258 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:49:08.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:49:08.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjz+lmNLFwOhAAlfrgRKXLKIHA4TIVXF4H4Q== --osd-uuid 9e856d44-dffd-48e9-94ec-a725f24fee80 2024-09-17T18:49:08.306 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:08.305+0000 7f293d9fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:08.326 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:08.325+0000 7f293d9fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:08.327 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:08.325+0000 7f293d9fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:08.327 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:08.325+0000 7f293d9fc7c0 -1 bdev(0x5557e716bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:49:08.328 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:08.325+0000 7f293d9fc7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:49:14.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:49:14.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:49:14.600 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:49:14.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:49:14.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:49:14.990 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:49:14.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:49:14.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:49:14.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:49:14.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:49:14.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:49:15.023 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:15.021+0000 7ff96da117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:15.048 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:15.045+0000 7ff96da117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:15.049 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:15.049+0000 7ff96da117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:15.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:49:15.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:49:15.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:49:15.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:49:15.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:49:15.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:15.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:49:15.330 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:49:15.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:15.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:15.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:16.332 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:16.329+0000 7ff96da117c0 -1 Falling back to public interface 2024-09-17T18:49:16.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:16.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:16.662 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:49:16.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:49:16.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:16.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:16.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:17.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:17.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:17.981 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:49:17.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:49:17.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:17.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:18.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:19.270 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:19.269+0000 7ff96da117c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:49:19.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:19.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:19.314 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:49:19.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:49:19.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:19.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:19.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:20.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:20.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:20.729 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:49:20.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:49:20.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:20.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:21.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:22.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:22.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:22.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:49:22.072 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:49:22.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:22.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:22.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:23.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:23.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:23.397 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:49:23.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:49:23.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:23.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:23.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:24.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:24.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:24.733 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:49:24.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:49:24.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:24.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:25.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:26.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:26.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:26.079 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:49:26.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:49:26.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:26.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:26.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:27.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:27.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:27.406 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:49:27.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:49:27.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:27.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:27.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:28.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:28.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:28.726 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:49:28.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:49:28.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:28.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:29.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:30.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:30.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:30.071 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:49:30.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:49:30.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:30.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:30.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:31.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:31.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:31.408 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:49:31.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:49:31.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:31.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:32.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:33.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:33.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:33.145 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:49:33.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:49:33.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:33.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:33.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:34.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:34.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:34.983 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:49:34.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:49:34.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:34.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:35.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:36.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:36.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:36.763 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:49:36.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:49:36.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:36.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:37.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:38.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:38.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:38.160 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:49:38.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:49:38.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:38.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:38.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:39.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:39.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:39.982 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:49:39.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:49:39.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:39.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:40.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:41.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:41.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:41.653 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:49:41.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:49:41.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:41.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:49:41.983 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1197198756,v1:127.0.0.1:6803/1197198756] [v2:127.0.0.1:6804/1197198756,v1:127.0.0.1:6805/1197198756] exists,up 9e856d44-dffd-48e9-94ec-a725f24fee80 2024-09-17T18:49:41.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:49:41.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:49:41.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:49:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:49:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:49:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:49:41.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:49:41.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:49:41.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:49:41.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:49:41.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 fb0396fc-e1f5-4859-87d9-e2730138c7fe' 2024-09-17T18:49:41.993 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:49:41.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:49:42.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBGz+lmXppOABAANT9czK4vNRtz+W+Fh1WfBg== 2024-09-17T18:49:42.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBGz+lmXppOABAANT9czK4vNRtz+W+Fh1WfBg=="}' 2024-09-17T18:49:42.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fb0396fc-e1f5-4859-87d9-e2730138c7fe -i td/osd-scrub-repair/1/new.json 2024-09-17T18:49:42.383 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:49:42.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:49:42.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGz+lmXppOABAANT9czK4vNRtz+W+Fh1WfBg== --osd-uuid fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:49:42.438 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:42.436+0000 7f849016c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:42.484 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:42.480+0000 7f849016c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:42.485 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:42.484+0000 7f849016c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:42.486 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:42.484+0000 7f849016c7c0 -1 bdev(0x5634ac863c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:49:42.486 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:42.484+0000 7f849016c7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:49:48.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:49:48.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:49:48.874 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:49:48.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:49:48.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:49:49.287 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:49:49.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:49:49.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:49:49.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:49:49.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:49:49.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:49:49.319 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:49.316+0000 7fe4c3d347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:49.346 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:49.344+0000 7fe4c3d347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:49.347 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:49.344+0000 7fe4c3d347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:49.619 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:49:49.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:49:49.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:49.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:49.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:50.848 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:50.844+0000 7fe4c3d347c0 -1 Falling back to public interface 2024-09-17T18:49:50.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:50.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:50.957 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:49:50.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:49:50.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:50.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:51.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:52.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:52.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:52.329 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:49:52.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:49:52.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:52.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:52.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:53.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:53.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:53.664 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:49:53.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:49:53.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:53.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:53.786 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:49:53.784+0000 7fe4c3d347c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:49:53.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:54.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:54.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:54.997 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:49:54.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:49:54.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:54.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:55.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:56.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:56.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:56.324 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:49:56.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:49:56.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:56.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:56.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:57.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:57.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:57.660 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:49:57.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:49:57.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:57.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:58.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:49:59.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:49:59.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:49:59.058 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:49:59.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:49:59.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:49:59.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:49:59.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:00.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:00.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:00.461 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:50:00.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:50:00.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:00.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:00.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:01.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:01.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:01.827 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:50:01.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:50:01.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:01.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:02.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:03.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:03.150 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:50:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:50:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:03.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:03.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:04.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:04.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:04.633 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:50:04.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:50:04.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:04.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:04.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:05.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:05.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:05.988 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:50:05.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:50:05.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:05.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:06.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:07.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:07.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:07.315 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:50:07.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:50:07.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:07.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:08.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:09.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:09.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:09.017 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:50:09.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:50:09.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:09.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:09.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:10.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:10.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:10.890 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:50:10.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:50:10.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:10.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:11.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:12.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:12.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:12.665 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:50:12.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:50:12.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:12.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:13.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:14.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:14.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:14.419 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:50:14.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:50:14.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:14.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:14.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:15.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:15.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:15.968 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:50:15.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:50:15.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:15.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:16.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:17.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:17.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:17.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:50:17.445 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:50:17.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:17.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:50:17.793 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3522865167,v1:127.0.0.1:6811/3522865167] [v2:127.0.0.1:6812/3522865167,v1:127.0.0.1:6813/3522865167] exists,up fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:50:17.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:50:17.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:50:17.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:50:17.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:50:17.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:50:17.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:50:17.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6959b9a6-c999-47fa-a4c3-2502d343297c 2024-09-17T18:50:17.800 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 6959b9a6-c999-47fa-a4c3-2502d343297c 2024-09-17T18:50:17.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6959b9a6-c999-47fa-a4c3-2502d343297c' 2024-09-17T18:50:17.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:50:17.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBpz+lmgG6WMBAALFJexAaSvVhyjPDJTVprFQ== 2024-09-17T18:50:17.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBpz+lmgG6WMBAALFJexAaSvVhyjPDJTVprFQ=="}' 2024-09-17T18:50:17.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6959b9a6-c999-47fa-a4c3-2502d343297c -i td/osd-scrub-repair/2/new.json 2024-09-17T18:50:18.202 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:50:18.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:50:18.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpz+lmgG6WMBAALFJexAaSvVhyjPDJTVprFQ== --osd-uuid 6959b9a6-c999-47fa-a4c3-2502d343297c 2024-09-17T18:50:18.256 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:18.251+0000 7fecc9dcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:18.295 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:18.291+0000 7fecc9dcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:18.297 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:18.291+0000 7fecc9dcd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:18.297 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:18.295+0000 7fecc9dcd7c0 -1 bdev(0x55b7594e3c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:50:18.297 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:18.295+0000 7fecc9dcd7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:50:24.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:50:24.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:50:24.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:50:24.710 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:50:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:50:25.136 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:50:25.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:50:25.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:50:25.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:50:25.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:50:25.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:50:25.172 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:25.167+0000 7faee79c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:25.172 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:25.167+0000 7faee79c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:25.173 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:25.171+0000 7faee79c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:25.492 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:50:25.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:50:25.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:25.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:25.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:26.660 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:26.655+0000 7faee79c37c0 -1 Falling back to public interface 2024-09-17T18:50:26.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:26.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:26.834 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:50:26.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:50:26.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:26.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:27.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:28.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:28.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:28.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:50:28.162 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:50:28.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:28.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:28.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:29.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:29.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:29.483 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:50:29.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:50:29.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:29.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:29.698 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:29.695+0000 7faee79c37c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:50:29.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:30.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:30.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:30.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:50:30.825 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:50:30.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:30.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:31.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:32.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:32.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:32.150 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:50:32.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:50:32.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:32.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:32.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:33.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:33.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:33.590 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:50:33.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:50:33.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:33.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:33.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:34.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:34.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:34.935 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:50:34.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:50:34.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:34.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:35.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:36.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:36.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:36.325 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:50:36.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:50:36.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:36.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:36.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:37.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:37.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:37.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:50:37.727 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:50:37.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:37.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:39.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:39.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:39.063 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:50:39.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:50:39.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:39.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:39.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:40.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:40.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:40.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:50:40.390 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:50:40.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:40.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:40.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:41.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:41.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:41.802 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:50:41.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:50:41.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:41.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:42.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:43.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:43.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:43.132 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:50:43.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:50:43.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:43.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:43.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:44.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:44.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:44.514 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:50:44.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:50:44.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:44.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:45.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:46.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:46.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:46.241 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:50:46.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:50:46.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:46.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:47.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:48.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:48.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:48.304 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:50:48.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:50:48.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:48.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:48.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:49.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:49.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:49.893 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:50:49.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:50:49.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:49.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:50.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:51.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:51.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:51.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:50:51.283 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:50:51.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:51.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:51.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:52.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:52.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:52.811 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:50:52.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:50:52.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:52.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:53.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:50:54.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:50:54.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:50:54.640 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:50:54.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:50:54.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:50:54.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:50:55.031 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1362500064,v1:127.0.0.1:6819/1362500064] [v2:127.0.0.1:6820/1362500064,v1:127.0.0.1:6821/1362500064] exists,up 6959b9a6-c999-47fa-a4c3-2502d343297c 2024-09-17T18:50:55.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:50:55.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:50:55.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:50:55.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:50:55.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-17T18:50:55.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:50:55.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:50:55.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:50:55.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:50:55.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:50:55.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:50:55.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 2450e678-f1ec-4f59-93f7-1797fa7877ab' 2024-09-17T18:50:55.039 INFO:tasks.workunit.client.0.smithi105.stdout:add osd3 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:50:55.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:50:55.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCPz+lmkR6vAxAAcdg2q/n0yjzgaj2GnjFeew== 2024-09-17T18:50:55.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCPz+lmkR6vAxAAcdg2q/n0yjzgaj2GnjFeew=="}' 2024-09-17T18:50:55.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2450e678-f1ec-4f59-93f7-1797fa7877ab -i td/osd-scrub-repair/3/new.json 2024-09-17T18:50:55.490 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:50:55.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-17T18:50:55.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPz+lmkR6vAxAAcdg2q/n0yjzgaj2GnjFeew== --osd-uuid 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:50:55.547 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:55.543+0000 7f0ae1f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:55.592 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:55.587+0000 7f0ae1f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:55.593 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:55.591+0000 7f0ae1f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:50:55.594 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:55.591+0000 7f0ae1f837c0 -1 bdev(0x55ba68cf7c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-17T18:50:55.594 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:50:55.591+0000 7f0ae1f837c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-17T18:51:02.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-17T18:51:02.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:51:02.730 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd3 key to auth repository 2024-09-17T18:51:02.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T18:51:02.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:51:03.140 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.3 2024-09-17T18:51:03.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T18:51:03.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:51:03.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:51:03.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:51:03.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:51:03.172 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:03.166+0000 7f4eed0ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:51:03.174 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:03.170+0000 7f4eed0ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:51:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:03.170+0000 7f4eed0ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:03.500 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:51:03.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:51:03.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:03.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:03.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:04.712 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:04.706+0000 7f4eed0ea7c0 -1 Falling back to public interface 2024-09-17T18:51:04.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:04.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:04.833 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:51:04.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:51:04.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:04.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:05.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:06.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:06.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:06.179 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:51:06.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:51:06.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:06.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:06.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:07.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:07.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:07.513 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:51:07.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:51:07.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:07.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:07.745 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:07.738+0000 7f4eed0ea7c0 -1 osd.3 0 log_to_monitors true 2024-09-17T18:51:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:08.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:08.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:08.845 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:51:08.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:51:08.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:08.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:09.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:10.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:10.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:10.163 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:51:10.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:51:10.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:10.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:10.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:11.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:11.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:11.585 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:51:11.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:51:11.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:11.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:11.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:12.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:12.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:12.974 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:51:12.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:51:12.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:12.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:14.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:14.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:14.304 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:51:14.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:51:14.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:14.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:14.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:15.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:15.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:15.642 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:51:15.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:51:15.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:15.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:15.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:16.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:16.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:16.975 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:51:16.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:51:16.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:16.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:17.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:18.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:18.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:18.330 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:51:18.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:51:18.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:18.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:18.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:19.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:19.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:19.819 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:51:19.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:51:19.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:19.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:20.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:21.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:21.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:21.155 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:51:21.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:51:21.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:21.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:21.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:22.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:22.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:22.834 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:51:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:51:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:22.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:23.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:24.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:24.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:24.585 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:51:24.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:51:24.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:24.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:25.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:26.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:26.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:26.163 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:51:26.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:51:26.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:26.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:26.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:27.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:27.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:27.705 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:51:27.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:51:27.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:27.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:28.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:29.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:29.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:29.718 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:51:29.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:51:29.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:29.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:30.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:31.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:31.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:31.407 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:51:31.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:51:31.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:31.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:31.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:32.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:32.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:32.959 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:51:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:51:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:33.334 INFO:tasks.workunit.client.0.smithi105.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3300893517,v1:127.0.0.1:6827/3300893517] [v2:127.0.0.1:6828/3300893517,v1:127.0.0.1:6829/3300893517] exists,up 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:51:33.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:51:33.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:51:33.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:51:33.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-17T18:51:33.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T18:51:33.771 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T18:51:33.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T18:51:33.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T18:51:34.236 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T18:51:34.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:51:35.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T18:51:37.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-17T18:51:37.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:51:37.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:51:37.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:51:37.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:51:37.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:51:37.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:51:37.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:51:37.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:51:37.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:51:37.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:51:37.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:51:37.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:51:37.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:51:37.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:51:37.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:51:38.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:51:38.249 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:51:38.249 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:51:38.250 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:51:38.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:51:38.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:38.250 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:51:38.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-17T18:51:38.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-17T18:51:38.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-17T18:51:38.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:51:38.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-17T18:51:38.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-17T18:51:38.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672984' 2024-09-17T18:51:38.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:38.560 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:51:38.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T18:51:38.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T18:51:38.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509455' 2024-09-17T18:51:38.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:38.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:51:38.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345930 2024-09-17T18:51:38.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345930 2024-09-17T18:51:38.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509455 3-85899345930' 2024-09-17T18:51:38.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:38.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-17T18:51:38.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:38.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:51:38.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-17T18:51:38.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:38.849 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836513 2024-09-17T18:51:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-17T18:51:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-17T18:51:38.849 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:51:39.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-17T18:51:39.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:39.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:39.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-17T18:51:39.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:51:39.200 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-17T18:51:39.200 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:39.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-17T18:51:39.201 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672984 2024-09-17T18:51:39.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-17T18:51:39.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:51:39.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-17T18:51:39.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:39.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-17T18:51:39.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:39.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:51:39.538 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-17T18:51:39.539 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:39.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T18:51:39.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-17T18:51:39.540 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509455 2024-09-17T18:51:39.540 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:51:39.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509455 2024-09-17T18:51:39.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:51:40.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:51:40.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:51:41.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509455 2024-09-17T18:51:41.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:41.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345930 2024-09-17T18:51:41.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:41.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:51:41.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345930 2024-09-17T18:51:41.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:41.227 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345930 2024-09-17T18:51:41.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345930 2024-09-17T18:51:41.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-17T18:51:41.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:51:41.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-17T18:51:41.587 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:51:41.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:51:41.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:51:41.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:51:41.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:51:41.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:51:41.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:51:41.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:51:41.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:51:41.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:51:41.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:51:42.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:51:42.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:51:42.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:51:42.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:51:42.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T18:51:42.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-17T18:51:43.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T18:51:43.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T18:51:44.381 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T18:51:44.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:51:45.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-17T18:51:45.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T18:51:45.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:51:45.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:51:45.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:51:45.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:51:45.407 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:51:45.407 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:51:45.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:51:45.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:51:45.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:51:45.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:51:45.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:51:45.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:51:45.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:51:45.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:51:45.533 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:51:45.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:51:45.964 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:51:45.964 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:51:45.964 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:51:45.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:51:45.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:45.965 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:51:46.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836515 2024-09-17T18:51:46.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836515 2024-09-17T18:51:46.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515' 2024-09-17T18:51:46.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:46.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:51:46.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672986 2024-09-17T18:51:46.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672986 2024-09-17T18:51:46.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672986' 2024-09-17T18:51:46.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:46.299 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:51:46.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-17T18:51:46.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-17T18:51:46.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509458' 2024-09-17T18:51:46.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:51:46.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:51:46.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345932 2024-09-17T18:51:46.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345932 2024-09-17T18:51:46.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509458 3-85899345932' 2024-09-17T18:51:46.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:46.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836515 2024-09-17T18:51:46.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:46.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:51:46.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836515 2024-09-17T18:51:46.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:46.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836515 2024-09-17T18:51:46.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-17T18:51:46.588 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836515 2024-09-17T18:51:46.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:51:46.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-17T18:51:46.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:46.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:46.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672986 2024-09-17T18:51:46.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:51:46.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672986 2024-09-17T18:51:46.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:46.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672986 2024-09-17T18:51:46.961 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672986 2024-09-17T18:51:46.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-17T18:51:46.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:51:47.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672986 -lt 42949672986 2024-09-17T18:51:47.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:47.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:47.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-17T18:51:47.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:51:47.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-17T18:51:47.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:47.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-17T18:51:47.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-17T18:51:47.309 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509458 2024-09-17T18:51:47.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:51:47.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-17T18:51:47.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:51:47.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345932 2024-09-17T18:51:47.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:51:47.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:51:47.643 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345932 2024-09-17T18:51:47.643 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:51:47.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345932 2024-09-17T18:51:47.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-09-17T18:51:47.645 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345932 2024-09-17T18:51:47.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:51:47.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-17T18:51:47.975 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:51:47.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:51:47.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:51:48.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:51:48.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:51:48.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:51:48.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:51:48.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:51:48.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:51:48.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:51:48.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:51:48.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:51:48.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:51:48.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:51:48.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T18:51:49.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T18:51:49.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-17T18:51:49.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T18:51:49.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T18:51:49.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T18:51:49.594 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T18:51:49.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T18:51:50.265 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T18:51:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T18:51:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T18:51:50.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:51:50.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-17T18:51:50.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T18:51:50.713 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:51:50.713 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:51:50.713 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:51:51.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-17T18:51:51.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-17T18:51:51.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-17T18:51:51.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T18:51:51.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T18:51:51.046 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:51:51.046 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T18:51:51.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T18:51:51.362 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:51:51.362 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:51:51.362 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T18:51:51.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:51:51.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:51:51.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:51:52.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T18:51:54.325 INFO:tasks.workunit.client.0.smithi105.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T18:51:54.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:51:54.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:51:54.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:51:54.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:51:54.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:51:54.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:51:54.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:51:54.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:51:54.871 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:51:54.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:51:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:51:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:51:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T18:51:54.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:51:54.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:51:54.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:51:54.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:51:54.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:51:54.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:51:54.907 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:54.905+0000 7fdec04a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:51:54.911 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:54.909+0000 7fdec04a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:51:54.912 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:54.909+0000 7fdec04a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:55.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:51:55.201 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:51:55.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:55.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:55.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:56.252 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:56.249+0000 7fdec04a57c0 -1 Falling back to public interface 2024-09-17T18:51:56.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:56.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:56.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:51:56.521 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:51:56.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:56.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:56.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:57.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:57.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:57.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:51:57.858 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:51:57.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:57.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:58.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:51:59.022 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:51:59.021+0000 7fdec04a57c0 -1 osd.3 37 log_to_monitors true 2024-09-17T18:51:59.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:51:59.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:51:59.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:51:59.181 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:51:59.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:51:59.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 41 up_thru 33 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/283575289,v1:127.0.0.1:6827/283575289] [v2:127.0.0.1:6828/283575289,v1:127.0.0.1:6829/283575289] exists,up 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:51:59.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:51:59.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:51:59.599 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:51:59.599 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:51:59.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:51:59.600 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:51:59.600 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:51:59.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:51:59.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:51:59.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:51:59.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:51:59.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:51:59.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:52:00.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:52:00.136 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:00.136 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:00.137 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:52:00.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:52:00.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:00.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:52:00.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836519 2024-09-17T18:52:00.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836519 2024-09-17T18:52:00.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519' 2024-09-17T18:52:00.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:00.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:52:00.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672990 2024-09-17T18:52:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672990 2024-09-17T18:52:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672990' 2024-09-17T18:52:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:00.598 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:52:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-17T18:52:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-17T18:52:00.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672990 2-64424509462' 2024-09-17T18:52:00.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:00.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:52:01.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-17T18:52:01.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-17T18:52:01.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672990 2-64424509462 3-176093659138' 2024-09-17T18:52:01.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:01.216 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836519 2024-09-17T18:52:01.216 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:01.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:52:01.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836519 2024-09-17T18:52:01.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:01.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836519 2024-09-17T18:52:01.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-17T18:52:01.219 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836519 2024-09-17T18:52:01.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:52:01.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836519 2024-09-17T18:52:01.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:01.556 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672990 2024-09-17T18:52:01.556 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:01.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:52:01.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672990 2024-09-17T18:52:01.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:01.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672990 2024-09-17T18:52:01.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-17T18:52:01.560 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672990 2024-09-17T18:52:01.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:52:01.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672990 2024-09-17T18:52:01.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:01.905 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-17T18:52:01.906 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:01.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:52:01.908 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-17T18:52:01.908 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:01.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-17T18:52:01.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-17T18:52:01.909 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509462 2024-09-17T18:52:01.909 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:52:02.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509461 -lt 64424509462 2024-09-17T18:52:02.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:52:03.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:52:03.288 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:52:03.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-17T18:52:03.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:03.606 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-176093659138 2024-09-17T18:52:03.606 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:03.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:52:03.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-176093659138 2024-09-17T18:52:03.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-17T18:52:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-09-17T18:52:03.611 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 176093659138 2024-09-17T18:52:03.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:52:03.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-17T18:52:03.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:52:03.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:03.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:04.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:52:04.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:52:04.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:52:04.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:52:04.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:52:04.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:52:04.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:52:04.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:52:04.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:52:04.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:52:04.676 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:04.676 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:05.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:52:05.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:52:05.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:52:05.057 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T18:52:05.057 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:52:05.057 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:52:05.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:52:05.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:52:05.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T18:52:05.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T18:52:05.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:52:05.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:52:05.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:05.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:05.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:05.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:05.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:05.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:52:06.026 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:52:06.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:06.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:52:06.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:06.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:52:06.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:52:06.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:06.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:06.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:06.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:06.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:06.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:06.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:51:44.135821+0000 '>' 2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:06.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:07.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:07.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:07.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:07.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:07.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:07.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:07.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:07.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:51:44.135821+0000 '>' 2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:07.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:08.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:08.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:08.733 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:08.733 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:08.733 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:08.734 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:08.734 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:09.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:51:44.135821+0000 '>' 2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:09.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:10.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:10.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:10.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:10.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:10.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:10.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:10.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:10.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:51:44.135821+0000 '>' 2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:10.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:11.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:11.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:11.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:11.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:11.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:11.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:11.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:11.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:06.112253+0000 '>' 2024-09-17T18:51:44.135821+0000 2024-09-17T18:52:11.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:52:11.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:52:13.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:52:13.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:52:13.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:52:13.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:52:13.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:52:13.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:52:13.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:52:14.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T18:52:14.519 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T18:52:14.519 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:52:14.519 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T18:52:14.519 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:52:14.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:52:14.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:52:14.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:52:14.806 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:52:14.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:52:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:52:14.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:52:14.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:52:14.810 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T18:52:14.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:52:14.810 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:52:14.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:52:14.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:52:14.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:52:14.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:52:14.843 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:14.841+0000 7fc1675a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:14.876 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:14.873+0000 7fc1675a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:14.877 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:14.877+0000 7fc1675a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:15.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:52:15.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:52:15.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:15.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:15.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:15.932 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:15.929+0000 7fc1675a87c0 -1 Falling back to public interface 2024-09-17T18:52:16.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:16.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:16.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:52:16.532 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:16.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:16.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:16.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:17.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:17.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:17.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:52:17.865 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:17.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:17.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:19.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:19.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:19.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:52:19.183 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:52:19.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:19.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:19.222 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:19.221+0000 7fc1675a87c0 -1 osd.3 43 log_to_monitors true 2024-09-17T18:52:19.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:20.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:20.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:20.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:52:20.546 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:52:20.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:20.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 47 up_thru 47 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/2601829259,v1:127.0.0.1:6827/2601829259] [v2:127.0.0.1:6828/2601829259,v1:127.0.0.1:6829/2601829259] exists,up 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:52:21.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:52:21.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:52:21.088 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:52:21.088 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:52:21.089 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:52:21.089 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:52:21.089 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:52:21.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:52:21.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:52:21.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:52:21.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:52:21.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:52:21.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:52:21.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:52:21.637 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:21.637 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:21.637 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:52:21.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:52:21.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:21.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:52:21.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-17T18:52:21.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-17T18:52:21.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-17T18:52:21.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:21.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:52:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672995 2024-09-17T18:52:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672995 2024-09-17T18:52:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672995' 2024-09-17T18:52:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:21.954 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:52:22.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-17T18:52:22.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-17T18:52:22.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672995 2-64424509467' 2024-09-17T18:52:22.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:22.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:52:22.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462914 2024-09-17T18:52:22.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462914 2024-09-17T18:52:22.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672995 2-64424509467 3-201863462914' 2024-09-17T18:52:22.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:22.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:22.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-17T18:52:22.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:52:22.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-17T18:52:22.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-17T18:52:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-17T18:52:22.246 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836524 2024-09-17T18:52:22.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:52:22.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-17T18:52:22.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:52:23.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:52:23.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:52:23.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-17T18:52:23.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:23.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672995 2024-09-17T18:52:23.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:23.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:52:23.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:23.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672995 2024-09-17T18:52:23.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672995 2024-09-17T18:52:23.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-09-17T18:52:23.893 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672995 2024-09-17T18:52:23.893 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:52:24.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672995 -lt 42949672995 2024-09-17T18:52:24.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:24.229 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-17T18:52:24.229 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:24.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:52:24.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-17T18:52:24.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-17T18:52:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-17T18:52:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509467 2024-09-17T18:52:24.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:52:24.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-17T18:52:24.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:24.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-201863462914 2024-09-17T18:52:24.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:24.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:52:24.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-201863462914 2024-09-17T18:52:24.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:24.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462914 2024-09-17T18:52:24.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-09-17T18:52:24.565 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 201863462914 2024-09-17T18:52:24.565 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:52:24.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462915 -lt 201863462914 2024-09-17T18:52:24.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:52:24.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:24.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:25.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:52:25.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:52:25.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:52:25.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:52:25.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:52:25.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:52:25.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:52:25.262 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:52:25.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:52:25.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:52:25.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:25.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:25.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:52:25.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:52:25.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:52:25.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:52:26.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:52:26.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-17T18:52:26.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T18:52:26.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T18:52:26.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:52:26.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:52:26.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:52:26.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:52:26.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:52:26.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:52:26.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:52:26.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:52:28.379 INFO:tasks.workunit.client.0.smithi105.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:52:28.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:52:28.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:52:28.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:52:28.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:52:28.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:52:28.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:52:28.917 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:52:28.917 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:52:28.917 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:52:28.917 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:52:28.917 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:52:28.918 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:52:28.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:52:28.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:52:28.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:52:28.920 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:52:28.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:52:28.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:52:28.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:52:28.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:52:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:52:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:52:28.954 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:28.953+0000 7f7b38e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:28.956 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:28.953+0000 7f7b38e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:28.957+0000 7f7b38e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:29.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:52:29.257 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:52:29.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:29.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:52:29.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:30.032 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:30.029+0000 7f7b38e347c0 -1 Falling back to public interface 2024-09-17T18:52:30.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:30.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:30.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:52:30.596 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:30.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:30.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:52:30.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:31.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:31.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:31.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:52:31.931 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:31.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:52:31.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:32.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:32.867 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:32.865+0000 7f7b38e347c0 -1 osd.1 49 log_to_monitors true 2024-09-17T18:52:33.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:33.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:33.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:52:33.255 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:52:33.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:33.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:52:33.713 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 54 up_thru 44 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/1484632760,v1:127.0.0.1:6811/1484632760] [v2:127.0.0.1:6812/1484632760,v1:127.0.0.1:6813/1484632760] exists,up fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:52:33.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:52:33.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:52:33.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:52:33.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:52:33.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:52:33.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:52:33.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:52:33.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:52:33.714 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:52:33.714 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:52:33.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:52:33.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:52:33.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:52:33.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:52:33.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:52:33.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:52:33.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:52:33.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:52:33.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:52:34.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:52:34.159 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:34.159 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:34.160 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:52:34.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:52:34.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:34.160 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:52:34.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836528 2024-09-17T18:52:34.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836528 2024-09-17T18:52:34.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836528' 2024-09-17T18:52:34.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:34.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:52:34.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=231928233986 2024-09-17T18:52:34.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 231928233986 2024-09-17T18:52:34.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836528 1-231928233986' 2024-09-17T18:52:34.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:34.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:52:34.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509470 2024-09-17T18:52:34.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509470 2024-09-17T18:52:34.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836528 1-231928233986 2-64424509470' 2024-09-17T18:52:34.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:34.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:52:35.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=201863462918 2024-09-17T18:52:35.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 201863462918 2024-09-17T18:52:35.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836528 1-231928233986 2-64424509470 3-201863462918' 2024-09-17T18:52:35.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:35.073 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836528 2024-09-17T18:52:35.073 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:35.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:52:35.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836528 2024-09-17T18:52:35.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:35.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836528 2024-09-17T18:52:35.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-09-17T18:52:35.076 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836528 2024-09-17T18:52:35.076 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:52:35.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836528 -lt 21474836528 2024-09-17T18:52:35.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:35.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-231928233986 2024-09-17T18:52:35.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:35.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:52:35.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-231928233986 2024-09-17T18:52:35.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:35.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=231928233986 2024-09-17T18:52:35.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-09-17T18:52:35.441 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 231928233986 2024-09-17T18:52:35.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:52:35.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 231928233986 -lt 231928233986 2024-09-17T18:52:35.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:35.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509470 2024-09-17T18:52:35.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:35.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:52:35.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509470 2024-09-17T18:52:35.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:35.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509470 2024-09-17T18:52:35.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-17T18:52:35.778 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509470 2024-09-17T18:52:35.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:52:36.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509469 -lt 64424509470 2024-09-17T18:52:36.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:52:37.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:52:37.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:52:37.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509471 -lt 64424509470 2024-09-17T18:52:37.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:37.446 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-201863462918 2024-09-17T18:52:37.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:37.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:52:37.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-201863462918 2024-09-17T18:52:37.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:37.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=201863462918 2024-09-17T18:52:37.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 201863462918' 2024-09-17T18:52:37.451 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 201863462918 2024-09-17T18:52:37.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:52:37.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 201863462918 -lt 201863462918 2024-09-17T18:52:37.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:52:37.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:37.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:38.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:52:38.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:52:38.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:52:38.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:52:38.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:52:38.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:52:38.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:52:38.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:52:38.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-17T18:52:38.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:52:38.533 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:38.533 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 5 2024-09-17T18:52:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-17T18:52:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:52:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-17T18:52:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:52:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:52:39.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:52:39.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:52:39.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:52:39.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:52:39.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:52:39.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:52:39.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:52:39.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:52:39.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:52:39.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:52:39.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:39.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:39.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:52:39.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:52:39.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:52:39.725 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T18:52:39.725 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:52:39.725 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:52:39.725 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:52:39.725 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:52:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T18:52:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T18:52:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:52:40.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:52:40.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:40.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:40.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:40.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:40.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:40.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:52:40.633 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:52:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:52:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:52:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:52:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:40.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:40.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:40.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:40.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:40.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:40.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:06.112253+0000 '>' 2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:40.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:41.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:41.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:41.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:41.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:41.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:41.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:41.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:42.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:06.112253+0000 '>' 2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:42.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:43.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:43.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:43.310 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:43.310 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:43.310 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:43.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:43.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:43.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:06.112253+0000 '>' 2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:43.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:52:44.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:52:44.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:52:44.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:52:44.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:52:44.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:52:44.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:52:44.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:52:44.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:41.211224+0000 '>' 2024-09-17T18:52:06.112253+0000 2024-09-17T18:52:44.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:52:44.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:52:46.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:52:46.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:52:46.972 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:52:46.972 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:52:46.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:52:46.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:52:46.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:52:47.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:52:47.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T18:52:47.773 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T18:52:47.773 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:52:47.773 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T18:52:47.773 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:52:48.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:52:48.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:52:48.075 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:52:48.076 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:52:48.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:52:48.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:52:48.078 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T18:52:48.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:52:48.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:52:48.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:52:48.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:52:48.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:52:48.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:52:48.112 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:48.108+0000 7f9f0c9ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:48.114 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:48.112+0000 7f9f0c9ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:48.116 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:48.112+0000 7f9f0c9ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:52:48.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:48.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:48.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:49.428 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:49.424+0000 7f9f0c9ee7c0 -1 Falling back to public interface 2024-09-17T18:52:49.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:49.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:49.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:52:49.744 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:49.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:49.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:50.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:51.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:51.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:51.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:52:51.092 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:51.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:51.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:51.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:52.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:52.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:52.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:52:52.417 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:52:52.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:52.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:52.659 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:52:52.656+0000 7f9f0c9ee7c0 -1 osd.3 56 log_to_monitors true 2024-09-17T18:52:52.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:52:53.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:52:53.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:52:53.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:52:53.769 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:52:53.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:52:53.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:52:54.142 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/4026301245,v1:127.0.0.1:6827/4026301245] [v2:127.0.0.1:6828/4026301245,v1:127.0.0.1:6829/4026301245] exists,up 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:52:54.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:52:54.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:52:54.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:52:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:52:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:52:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:52:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:52:54.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:52:54.143 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:52:54.144 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:52:54.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:52:54.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:52:54.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:52:54.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:52:54.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:52:54.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:52:54.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:52:54.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:52:54.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:52:54.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:52:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:52:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:52:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:52:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:52:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:54.629 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:52:54.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-17T18:52:54.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-17T18:52:54.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-17T18:52:54.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:54.773 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:52:54.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=231928233991 2024-09-17T18:52:54.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 231928233991 2024-09-17T18:52:54.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-231928233991' 2024-09-17T18:52:54.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:54.904 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:52:55.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509475 2024-09-17T18:52:55.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509475 2024-09-17T18:52:55.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-231928233991 2-64424509475' 2024-09-17T18:52:55.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:52:55.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:52:55.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037762 2024-09-17T18:52:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037762 2024-09-17T18:52:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-231928233991 2-64424509475 3-257698037762' 2024-09-17T18:52:55.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:55.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-17T18:52:55.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:55.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:52:55.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-17T18:52:55.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:55.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-17T18:52:55.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-17T18:52:55.209 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836533 2024-09-17T18:52:55.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:52:55.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836533 2024-09-17T18:52:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:52:56.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:52:56.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:52:56.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836533 2024-09-17T18:52:56.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:56.833 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-231928233991 2024-09-17T18:52:56.833 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:56.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:52:56.835 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-231928233991 2024-09-17T18:52:56.835 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:56.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=231928233991 2024-09-17T18:52:56.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 231928233991' 2024-09-17T18:52:56.837 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 231928233991 2024-09-17T18:52:56.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:52:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 231928233991 -lt 231928233991 2024-09-17T18:52:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:57.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509475 2024-09-17T18:52:57.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:57.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:52:57.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509475 2024-09-17T18:52:57.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:57.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509475 2024-09-17T18:52:57.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-17T18:52:57.206 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509475 2024-09-17T18:52:57.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:52:57.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509476 -lt 64424509475 2024-09-17T18:52:57.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:52:57.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037762 2024-09-17T18:52:57.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:52:57.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:52:57.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037762 2024-09-17T18:52:57.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:52:57.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037762 2024-09-17T18:52:57.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-09-17T18:52:57.569 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 257698037762 2024-09-17T18:52:57.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:52:57.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037762 -lt 257698037762 2024-09-17T18:52:57.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:52:57.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:57.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:58.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:52:58.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:52:58.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:52:58.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:52:58.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:52:58.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:52:58.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:52:58.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:52:58.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:52:58.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:52:58.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:52:58.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:52:59.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:52:59.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:52:59.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:52:59.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:52:59.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:52:59.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 142053"' 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 142053' 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 142054"' 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 142054' 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 142053 142054' 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 142053 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:52:59.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/142056: /' 2024-09-17T18:52:59.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/142058: /' 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:53:03.085 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:53:03.086 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:53:03.087 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:53:03.091 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:53:03.092 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:53:03.093 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:53:03.093 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:53:03.093 INFO:tasks.workunit.client.0.smithi105.stderr:142056: start osd.1 2024-09-17T18:53:03.093 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:1142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:53:03.233 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-17T18:53:03.234 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:53:03.235 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:53:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:53:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:53:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:53:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:53:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:53:03.236 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:53:03.237 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: start osd.2 2024-09-17T18:53:03.238 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:53:07.971 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T18:53:07.971 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:53:07.971 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:53:07.971 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2024-09-17T18:53:03.260+0000 7f6b0d6187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2024-09-17T18:53:03.264+0000 7f6b0d6187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2024-09-17T18:53:03.264+0000 7f6b0d6187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 0 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:07.972 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2024-09-17T18:53:04.600+0000 7f6b0d6187c0 -1 Falling back to public interface 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 1 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2024-09-17T18:53:07.444+0000 7f6b0d6187c0 -1 osd.2 62 log_to_monitors true 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 3 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:07.974 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:08.604 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers42056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:53:08.604 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:53:08.604 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:53:08.604 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:53:08.604 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:53:08.604 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2024-09-17T18:53:03.120+0000 7f2c5ab6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2024-09-17T18:53:03.156+0000 7f2c5ab6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2024-09-17T18:53:03.156+0000 7f2c5ab6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 0 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:08.605 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 1 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2024-09-17T18:53:04.980+0000 7f2c5ab6c7c0 -1 Falling back to public interface 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:53:08.606 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 3 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:08.607 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:10.112 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2024-09-17T18:53:08.600+0000 7f2c5ab6c7c0 -.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 4 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: osd.2 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [15,62) [v2:127.0.0.1:6810/999140764,v1:127.0.0.1:6811/999140764] [v2:127.0.0.1:6812/999140764,v1:127.0.0.1:6813/999140764] exists,up 6959b9a6-c999-47fa-a4c3-2502d343297c 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:53:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:53:10.114 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 1 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 2 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: 3' 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836537 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836537 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537' 2024-09-17T18:53:10.115 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:10.902 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.1 osd.1 62 log_to_monitors true 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 4 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:53:10.903 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 5 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: osd.1 up in weight 1 up_from 70 up_thru 57 down_at 63 last_clean_interval [54,62) [v2:127.0.0.1:6818/1326057026,v1:127.0.0.1:6819/1326057026] [v2:127.0.0.1:6820/1326057026,v1:127.0.0.1:6821/1326057026] exists,up fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:53:10.904 INFO:tasks.workunit.client.0.smithi105.stderr:142056: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 1 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 2 2024-09-17T18:53:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:142056: 3' 2024-09-17T18:53:10.906 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:53:10.906 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ush:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710722 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710722 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-300647710722' 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841538 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841538 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-300647710722 2-283467841538' 2024-09-17T18:53:12.490 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037766 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037766 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-300647710722 2-283467841538 3-257698037766' 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836537 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836537 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836537 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-09-17T18:53:12.491 INFO:tasks.workunit.client.0.smithi105.stderr:142058: waiting osd.0 seq 21474836537 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836536 -lt 21474836537 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836537 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-300647710722 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-300647710722 2024-09-17T18:53:12.492 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:13.691 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:53:13.691 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836538 2024-09-17T18:53:13.691 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836538 2024-09-17T18:53:13.691 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538' 2024-09-17T18:53:13.691 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=300647710723 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 300647710723 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-300647710723' 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=283467841539 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 283467841539 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-300647710723 2-283467841539' 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037767 2024-09-17T18:53:13.692 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037767 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836538 1-300647710723 2-283467841539 3-257698037767' 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836538 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836538 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836538 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: waiting osd.0 seq 21474836538 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836537 -lt 21474836538 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:53:13.693 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:53:13.694 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:13.694 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836538 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa//cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710722 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 300647710722' 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: waiting osd.1 seq 300647710722 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710723 -lt 300647710722 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-283467841538 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-283467841538 2024-09-17T18:53:13.954 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841538 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 283467841538' 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: waiting osd.2 seq 283467841538 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841539 -lt 283467841538 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037766 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037766 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037766 2024-09-17T18:53:13.955 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: waiting osd.3 seq 257698037766 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037767 -lt 257698037766 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:13.956 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stalestandalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-300647710723 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-300647710723 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=300647710723 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 300647710723' 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: waiting osd.1 seq 300647710723 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:53:15.122 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 300647710723 -lt 300647710723 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-283467841539 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-283467841539 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=283467841539 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: waiting osd.2 seq 283467841539 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 283467841539 -lt 283467841539 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037767 2024-09-17T18:53:15.123 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037767 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037767 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037767' 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: waiting osd.3 seq 257698037767 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037767 -lt 257698037767 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:15.124 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142") | not)' 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:53:15.431 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:15.433 INFO:tasks.workunit.client.0.smithi105.stderr:142058: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:15.433 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:53:15.433 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:53:15.433 INFO:tasks.workunit.client.0.smithi105.stderr:142058: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:53:15.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:53:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:142056: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:53:15.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 142054 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:53:15.927 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:53:15.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:53:15.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:53:16.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T18:53:16.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T18:53:16.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:53:16.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:53:16.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:16.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:16.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:16.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:16.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:52:41.211224+0000 2024-09-17T18:53:16.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:53:16.850 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:53:16.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:52:41.211224+0000 2024-09-17T18:53:16.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:53:16.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:52:41.211224+0000 2024-09-17T18:53:16.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:53:16.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:53:16.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:53:16.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:53:16.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:16.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:16.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:16.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:17.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:41.211224+0000 '>' 2024-09-17T18:52:41.211224+0000 2024-09-17T18:53:17.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:53:18.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:53:18.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:53:18.180 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:53:18.180 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:18.180 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:18.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:18.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:18.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:52:41.211224+0000 '>' 2024-09-17T18:52:41.211224+0000 2024-09-17T18:53:18.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:53:19.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:53:19.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:53:19.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:53:19.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:19.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:19.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:19.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:19.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:53:17.572346+0000 '>' 2024-09-17T18:52:41.211224+0000 2024-09-17T18:53:19.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:53:19.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:53:21.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T18:53:21.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:53:21.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:53:21.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:53:21.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 144999"' 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 144999' 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 145002"' 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 145002' 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/145001: /' 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 144999 145002' 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:53:21.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 144999 2024-09-17T18:53:21.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:53:21.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/145004: /' 2024-09-17T18:53:23.016 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.016 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:53:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: _ 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: hinfo_key 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: snapset 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:53:23.018 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:53:23.019 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:53:23.019 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:53:23.019 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:53:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:53:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:53:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:53:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:53:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:53:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:145001: start osd.1 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:53:23.062 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: _ 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: hinfo_key 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: snapset 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:53:23.063 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:53:23.064 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:53:23.064 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:53:23.064 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:53:23.064 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:53:23.064 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:53:23.065 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:53:23.065 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:53:23.065 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:53:23.065 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:53:23.066 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:53:23.067 INFO:tasks.workunit.client.0.smithi105.stderr:145004: start osd.2 2024-09-17T18:53:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:53:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:53:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:53:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:53:27.855 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2024-09-17T18:53:23.060+0000 7f0fb59d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2024-09-17T18:53:23.188+0000 7f0fb59d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2024-09-17T18:53:23.188+0000 7f0fb59d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 0 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:27.856 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2024-09-17T18:53:23.984+0000 7f0fb59d67c0 -1 Falling back to public interface 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 1 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:53:27.857 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 3 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:27.858 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2024-09-17T18:53:27.511+0000 7f0fb59d67c0 -1 osd.1 73 log_to_monitors true 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cevery_ops 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:53:28.828 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2024-09-17T18:53:23.092+0000 7ff953ee97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2024-09-17T18:53:23.188+0000 7ff953ee97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2024-09-17T18:53:23.188+0000 7ff953ee97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 0 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:28.829 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2024-09-17T18:53:24.692+0000 7ff953ee97c0 -1 Falling back to public interface 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 1 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2 2024-09-17T18:53:28.830 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 3 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:28.831 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.clienph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 4 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: osd.1 up in weight 1 up_from 78 up_thru 78 down_at 74 last_clean_interval [70,73) [v2:127.0.0.1:6810/4051045989,v1:127.0.0.1:6811/4051045989] [v2:127.0.0.1:6812/4051045989,v1:127.0.0.1:6813/4051045989] exists,up fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:53:30.028 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:53:30.029 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 1 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 2 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: 3' 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-17T18:53:30.030 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:31.543 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:53:31.543 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449090 2024-09-17T18:53:31.543 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449090 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-335007449090' 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-335007449090 2-352187318274' 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037772 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037772 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-335007449090 2-352187318274 3-257698037772' 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:31.544 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: waiting osd.0 seq 21474836543 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836543 -lt 21474836543 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-335007449090 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:53:31.545 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-335007449090 2024-09-17T18:53:31.546 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:31.546 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449090 2024-09-17T18:53:31.546 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 335007449090' 2024-09-17T18:53:31.546 INFO:tasks.workunit.client.0.smithi105.stderr:145001: waiting osd.1 seq 335007449090 2024-09-17T18:53:31.546 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:53:31.548 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpet.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:31.548 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:31.548 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 4 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2024-09-17T18:53:28.867+0000 7ff953ee97c0 -1 osd.2 73 log_to_monitors true 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 5 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: osd.2 up in weight 1 up_from 82 up_thru 66 down_at 74 last_clean_interval [66,73) [v2:127.0.0.1:6818/38259350,v1:127.0.0.1:6819/38259350] [v2:127.0.0.1:6820/38259350,v1:127.0.0.1:6821/38259350] exists,up 6959b9a6-c999-47fa-a4c3-2502d343297c 2024-09-17T18:53:31.549 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:53:31.550 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 1 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 2 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: 3' 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:53:31.551 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836544 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836544 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544' 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=335007449091 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 335007449091 2024-09-17T18:53:33.746 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-335007449091' 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318275 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318275 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-335007449091 2-352187318275' 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=257698037773 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 257698037773 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836544 1-335007449091 2-352187318275 3-257698037773' 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836544 2024-09-17T18:53:33.747 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836544 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836544 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: waiting osd.0 seq 21474836544 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836543 -lt 21474836544 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:33.748 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836544 2024-09-17T18:53:33.969 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qrs.sh:2222: flush_pg_stats: test 335007449090 -lt 335007449090 2024-09-17T18:53:33.969 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:33.969 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318274 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318274 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: waiting osd.2 seq 352187318274 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037772 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:33.970 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037772 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037772 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: waiting osd.3 seq 257698037772 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037771 -lt 257698037772 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037773 -lt 257698037772 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:33.971 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:53:33.972 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:33.972 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:33.972 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:33.972 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:34.700 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:145001: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:53:34.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 145002 2024-09-17T18:53:35.116 INFO:tasks.workunit.client.0.smithi105.stderr:a/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-335007449091 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-335007449091 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=335007449091 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 335007449091' 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: waiting osd.1 seq 335007449091 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 335007449091 -lt 335007449091 2024-09-17T18:53:35.117 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318275 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318275 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318275 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318275' 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: waiting osd.2 seq 352187318275 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318275 -lt 352187318275 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-257698037773 2024-09-17T18:53:35.118 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-257698037773 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=257698037773 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: waiting osd.3 seq 257698037773 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 257698037773 -lt 257698037773 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:35.119 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:53:35.806 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:35.806 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:35.806 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:35.806 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:35.806 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:35.806 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:145004: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:53:35.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:53:35.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:53:35.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T18:53:35.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:53:35.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:53:35.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T18:53:35.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 147404"' 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 147404' 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 147406"' 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 147406' 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 147404 147406' 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:53:35.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:53:35.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 147404 2024-09-17T18:53:35.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/147407: /' 2024-09-17T18:53:35.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:53:35.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/147409: /' 2024-09-17T18:53:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:53:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:53:39.292 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:53:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:53:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:53:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:53:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:53:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:53:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:53:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:53:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:53:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:53:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:53:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:147407: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:53:39.299 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:53:39.300 INFO:tasks.workunit.client.0.smithi105.stderr:147407: start osd.3 2024-09-17T18:53:39.300 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:1147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:53:39.580 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:53:39.581 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:53:39.582 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:53:39.584 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:53:39.584 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:53:39.584 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:53:39.584 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:53:39.584 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:53:39.585 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:53:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:147409: start osd.1 2024-09-17T18:53:39.587 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:53:44.071 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2024-09-17T18:53:39.335+0000 7efff1bb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2024-09-17T18:53:39.335+0000 7efff1bb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2024-09-17T18:53:39.339+0000 7efff1bb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:53:44.072 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 0 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2024-09-17T18:53:40.895+0000 7efff1bb17c0 -1 Falling back to public interface 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 1 2024-09-17T18:53:44.073 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 3 2024-09-17T18:53:44.074 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.075 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:53:44.075 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2024-09-17T18:53:43.739+0000 7efff1bb17c0 -1 osd.3 83 log_to_monitors true 2024-09-17T18:53:44.788 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers47409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:53:44.788 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:53:44.788 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:53:44.788 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:53:44.788 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2024-09-17T18:53:39.615+0000 7f32a35dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2024-09-17T18:53:39.619+0000 7f32a35dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2024-09-17T18:53:39.619+0000 7f32a35dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 0 2024-09-17T18:53:44.789 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2024-09-17T18:53:41.159+0000 7f32a35dd7c0 -1 Falling back to public interface 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 1 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.790 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 3 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:44.791 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:46.285 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2024-09-17T18:53:44.783+0000 7f32a35dd7c0 -.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 4 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: osd.3 up in weight 1 up_from 88 up_thru 88 down_at 84 last_clean_interval [60,83) [v2:127.0.0.1:6810/938312336,v1:127.0.0.1:6811/938312336] [v2:127.0.0.1:6812/938312336,v1:127.0.0.1:6813/938312336] exists,up 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:53:46.286 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:53:46.287 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 1 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 2 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: 3' 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-17T18:53:46.288 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:47.558 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.1 osd.1 83 log_to_monitors true 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 4 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:53:47.559 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 5 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: osd.1 up in weight 1 up_from 92 up_thru 92 down_at 84 last_clean_interval [78,83) [v2:127.0.0.1:6826/1673269200,v1:127.0.0.1:6827/1673269200] [v2:127.0.0.1:6828/1673269200,v1:127.0.0.1:6829/1673269200] exists,up fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:53:47.560 INFO:tasks.workunit.client.0.smithi105.stderr:147409: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 1 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 2 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: 3' 2024-09-17T18:53:47.561 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:53:47.562 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ush:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=2 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 2 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-2' 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318279 2024-09-17T18:53:48.156 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318279 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-2 2-352187318279' 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122050 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122050 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-2 2-352187318279 3-377957122050' 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-17T18:53:48.157 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-17T18:53:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:147407: waiting osd.0 seq 21474836548 2024-09-17T18:53:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836548 2024-09-17T18:53:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-2 2024-09-17T18:53:48.158 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:48.160 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-2 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=2 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 2' 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: waiting osd.1 seq 2 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:53:48.161 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 2 -lt 2 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: fbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836550 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836550 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550' 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=395136991235 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 395136991235 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-395136991235' 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:53:49.962 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318280 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318280 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-395136991235 2-352187318280' 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122051 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122051 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836550 1-395136991235 2-352187318280 3-377957122051' 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836550 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836550 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:49.963 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836550 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: waiting osd.0 seq 21474836550 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836550 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:53:49.964 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836550 -lt 21474836550 2024-09-17T18:53:50.530 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/lush_pg_stats: for s in $seqs 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318279 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318279 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318279 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318279' 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: waiting osd.2 seq 352187318279 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318278 -lt 352187318279 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:53:50.531 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318280 -lt 352187318279 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122050 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122050 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122050 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122050' 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: waiting osd.3 seq 377957122050 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122051 -lt 377957122050 2024-09-17T18:53:50.532 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:50.533 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-395136991235 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-395136991235 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=395136991235 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 395136991235' 2024-09-17T18:53:51.383 INFO:tasks.workunit.client.0.smithi105.stderr:147409: waiting osd.1 seq 395136991235 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 395136991235 -lt 395136991235 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318280 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318280 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318280 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318280' 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: waiting osd.2 seq 352187318280 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:53:51.384 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318280 -lt 352187318280 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-377957122051 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-377957122051 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122051 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 377957122051' 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: waiting osd.3 seq 377957122051 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122052 -lt 377957122051 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:53:51.385 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:51.386 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:51.386 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:53:52.186 INFO:tasks.workunit.client.0.smithi105.stderr:147json pg dump pgs 2024-09-17T18:53:52.186 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:52.187 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:52.188 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:53:52.189 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:53:52.189 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:53:52.189 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:53:52.189 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:53:52.189 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:53:52.189 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:53:52.935 INFO:tasks.workunit.client.0.smithi105.stderr:14409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:52.935 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:52.935 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:52.935 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:52.935 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:52.936 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:52.937 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_7407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:53.817 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:53:53.818 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-17T18:53:53.819 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_cleanprogress 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-17T18:53:54.155 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:54.156 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:53:54.157 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:53:55.173 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_r: loop=0 2024-09-17T18:53:55.173 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:55.173 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:55.173 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:53:55.174 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=0 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 0 '!=' null 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:55.175 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:55.176 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cecovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=1 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 1 '!=' null 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:53:55.450 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:53:55.451 INFO:tasks.workunit.client.0.smithi105.stderr:147409: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:53:55.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:53:55.875 INFO:tasks.workunit.client.0.smithi105.stderr:ephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:53:55.876 INFO:tasks.workunit.client.0.smithi105.stderr:147407: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:53:55.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 147406 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T18:53:55.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:53:55.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:53:55.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:53:55.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:53:56.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:56.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:56.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:53:17.572346+0000 2024-09-17T18:53:56.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:53:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:53:56.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:53:17.572346+0000 2024-09-17T18:53:56.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:53:56.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:53:17.572346+0000 2024-09-17T18:53:56.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:53:56.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:53:56.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:53:56.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:53:56.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:56.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:56.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:56.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:53:17.572346+0000 '>' 2024-09-17T18:53:17.572346+0000 2024-09-17T18:53:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:53:58.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:53:58.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:53:58.200 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:53:58.200 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:58.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:58.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:58.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:58.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:53:17.572346+0000 '>' 2024-09-17T18:53:17.572346+0000 2024-09-17T18:53:58.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:53:59.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:53:59.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:53:59.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:53:59.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:53:59.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:53:59.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:53:59.532 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:53:59.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:53:17.572346+0000 '>' 2024-09-17T18:53:17.572346+0000 2024-09-17T18:53:59.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:54:00.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:54:00.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:54:00.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:54:00.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:54:00.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:54:00.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:54:00.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:54:01.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:53:56.998329+0000 '>' 2024-09-17T18:53:17.572346+0000 2024-09-17T18:54:01.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:54:01.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:54:03.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 150788"' 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 150788' 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:54:03.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:54:03.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 150790"' 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 150790' 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 150788 150790' 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 150788 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/150791: /' 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:54:03.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/150794: /' 2024-09-17T18:54:05.208 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.208 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:54:05.208 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:54:05.208 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:54:05.209 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: _ 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: hinfo_key 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: snapset 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:54:05.210 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:54:05.211 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:54:05.211 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:54:05.211 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:54:05.211 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:54:05.213 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:54:05.213 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:54:05.213 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:54:05.213 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:54:05.213 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:54:05.214 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:54:05.215 INFO:tasks.workunit.client.0.smithi105.stderr:150794: start osd.1 2024-09-17T18:54:05.226 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.226 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:54:05.226 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:54:05.226 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:54:05.226 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:54:05.226 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:54:05.227 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: _ 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: hinfo_key 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: snapset 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:54:05.228 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:54:05.229 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:54:05.230 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:54:05.230 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:54:05.230 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:54:05.230 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:54:05.230 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:54:05.231 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:54:05.232 INFO:tasks.workunit.client.0.smithi105.stderr:150791: start osd.3 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-lvery_ops 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2024-09-17T18:54:05.247+0000 7f12e06e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2024-09-17T18:54:05.251+0000 7f12e06e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2024-09-17T18:54:05.251+0000 7f12e06e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:09.985 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 0 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2024-09-17T18:54:06.323+0000 7f12e06e27c0 -1 Falling back to public interface 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:09.986 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 1 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:09.987 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 3 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2024-09-17T18:54:09.835+0000 7f12e06e27c0 -1 osd.1 95 log_to_monitors true 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:54:09.988 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2024-09-17T18:54:05.271+0000 7f3cd9c717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2024-09-17T18:54:05.275+0000 7f3cd9c717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2024-09-17T18:54:05.279+0000 7f3cd9c717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 0 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:09.989 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2024-09-17T18:54:06.779+0000 7f3cd9c717c0 -1 Falling back to public interface 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 1 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:54:09.990 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:09.991 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:09.991 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:09.991 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:54:09.991 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 3 2024-09-17T18:54:09.991 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:09.991 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:54:12.428 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.ph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 4 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: osd.1 up in weight 1 up_from 98 up_thru 98 down_at 95 last_clean_interval [92,95) [v2:127.0.0.1:6810/2215818199,v1:127.0.0.1:6811/2215818199] [v2:127.0.0.1:6812/2215818199,v1:127.0.0.1:6813/2215818199] exists,up fb0396fc-e1f5-4859-87d9-e2730138c7fe 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:54:12.429 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:54:12.430 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 1 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 2 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: 3' 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836555 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836555 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555' 2024-09-17T18:54:12.431 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 4 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2024-09-17T18:54:11.127+0000 7f3cd9c717c0 -1 osd.3 94 log_to_monitors true 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:13.423 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 5 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: osd.3 up in weight 1 up_from 101 up_thru 101 down_at 95 last_clean_interval [88,94) [v2:127.0.0.1:6826/3908500179,v1:127.0.0.1:6827/3908500179] [v2:127.0.0.1:6828/3908500179,v1:127.0.0.1:6829/3908500179] exists,up 2450e678-f1ec-4f59-93f7-1797fa7877ab 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:54:13.424 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 1 2024-09-17T18:54:13.425 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 2 2024-09-17T18:54:13.426 INFO:tasks.workunit.client.0.smithi105.stderr:150791: 3' 2024-09-17T18:54:13.426 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubu/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-420906795010' 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318286 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318286 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-420906795010 2-352187318286' 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:13.866 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696898 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696898 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-420906795010 2-352187318286 3-433791696898' 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836555 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836555 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836555 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: waiting osd.0 seq 21474836555 2024-09-17T18:54:13.867 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836555 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-420906795010 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-420906795010 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 420906795010' 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: waiting osd.1 seq 420906795010 2024-09-17T18:54:13.868 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpentu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836557 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836557 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557' 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795011 2024-09-17T18:54:15.850 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795011 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-420906795011' 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318287 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318287 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-420906795011 2-352187318287' 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696899 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696899 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836557 1-420906795011 2-352187318287 3-433791696899' 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:15.851 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836557 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836557 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836557 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: waiting osd.0 seq 21474836557 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836555 -lt 21474836557 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:54:15.852 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:54:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hers.sh:2222: flush_pg_stats: test 420906795010 -lt 420906795010 2024-09-17T18:54:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318286 2024-09-17T18:54:16.402 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318286 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318286 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318286' 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: waiting osd.2 seq 352187318286 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318285 -lt 352187318286 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318287 -lt 352187318286 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:16.403 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-433791696898 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-433791696898 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696898 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 433791696898' 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: waiting osd.3 seq 433791696898 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696899 -lt 433791696898 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:16.404 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:54:16.405 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:54:16.405 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:54:16.405 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:54:16.405 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='selectlpers.sh:2222: flush_pg_stats: test 21474836557 -lt 21474836557 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-420906795011 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-420906795011 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:17.261 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795011 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 420906795011' 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: waiting osd.1 seq 420906795011 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795011 -lt 420906795011 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318287 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318287 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318287 2024-09-17T18:54:17.262 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318287' 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: waiting osd.2 seq 352187318287 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318287 -lt 352187318287 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-433791696899 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-433791696899 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696899 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 433791696899' 2024-09-17T18:54:17.263 INFO:tasks.workunit.client.0.smithi105.stderr:150791: waiting osd.3 seq 433791696899 2024-09-17T18:54:17.264 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:54:17.264 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696899 -lt 433791696899 2024-09-17T18:54:17.264 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:54:17.264 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:17.264 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:17.881 INFO:tasks.workunit.client.0.smithi105.stderr:(contains("stale") | not)' 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T18:54:17.882 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:17.883 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:54:17.884 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:54:17.884 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:54:17.884 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:54:17.884 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:54:17.884 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:54:18.789 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_se150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:54:18.789 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:54:18.790 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:54:18.791 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:54:18.792 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:54:18.792 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:18.792 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:18.792 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T18:54:18.792 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T18:54:19.125 INFO:tasks.workunit.client.0.smithi105.stderr:150791: c 2024-09-17T18:54:19.125 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:54:19.125 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:54:19.126 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:150794: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:150794: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:54:19.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:54:20.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:54:20.116 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:54:20.117 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:150791: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:54:20.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 150790 2024-09-17T18:54:20.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:54:20.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:54:20.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:54:20.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T18:54:20.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:54:20.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:54:20.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:54:20.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:54:20.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:54:20.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:54:20.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:54:20.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:54:20.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:54:20.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:54:20.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:54:20.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:54:20.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:54:20.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:54:20.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:54:20.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:54:20.520 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:54:20.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:54:20.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:54:20.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:54:20.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:54:20.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:20.560 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:20.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:54:20.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:54:20.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:54:20.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:54:20.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:54:20.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:54:20.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:54:20.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:54:20.565 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:54:20.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:54:20.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:54:20.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:54:20.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:54:20.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:54:20.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:54:20.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:54:20.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:54:20.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:54:20.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:54:20.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:54:20.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:54:20.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:54:20.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:20.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:20.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:54:20.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:54:20.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:54:20.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:54:20.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:54:20.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:20.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:20.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:54:20.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:54:20.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:54:20.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:54:20.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:54:21.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:54:21.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:54:21.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:54:21.068 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:54:21.069 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:21.069 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:21.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:21.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:54:21.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:54:21.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:54:21.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:54:21.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:54:21.270 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:54:21.270 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:54:21.270 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:54:21.270 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:54:21.270 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:54:21.271 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:54:21.271 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:21.271 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:21.271 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:54:21.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:54:21.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:54:21.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:54:21.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:54:21.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:54:21.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:54:21.384 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:54:21.384 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:54:21.384 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:54:21.384 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:54:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:54:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:54:21.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:54:21.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:54:21.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:54:22.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:54:22.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:54:22.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:54:22.363 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:54:22.363 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:22.363 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:22.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:22.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:54:22.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:54:22.401 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:54:22.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:22.406 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:54:22.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:54:22.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:54:22.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5a73f560-d0f9-4229-b80c-05ebcaab8714 2024-09-17T18:54:22.408 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 5a73f560-d0f9-4229-b80c-05ebcaab8714 2024-09-17T18:54:22.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5a73f560-d0f9-4229-b80c-05ebcaab8714' 2024-09-17T18:54:22.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:54:22.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBe0Olmtfw5GRAAufpmgjKYAe6376T5UBIDYA== 2024-09-17T18:54:22.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBe0Olmtfw5GRAAufpmgjKYAe6376T5UBIDYA=="}' 2024-09-17T18:54:22.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5a73f560-d0f9-4229-b80c-05ebcaab8714 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:54:22.673 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:54:22.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:54:22.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBe0Olmtfw5GRAAufpmgjKYAe6376T5UBIDYA== --osd-uuid 5a73f560-d0f9-4229-b80c-05ebcaab8714 2024-09-17T18:54:22.717 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:22.710+0000 7f6dca6da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:22.732 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:22.726+0000 7f6dca6da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:22.733 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:22.726+0000 7f6dca6da7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:22.734 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:22.730+0000 7f6dca6da7c0 -1 bdev(0x55d3f4681c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:54:22.734 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:22.730+0000 7f6dca6da7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:54:29.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:54:29.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:54:29.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:54:29.032 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:54:29.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:54:29.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:54:29.452 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:54:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:54:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:54:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:54:29.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:54:29.485 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:29.478+0000 7fd8b02917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:29.513 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:29.506+0000 7fd8b02917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:29.514 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:29.510+0000 7fd8b02917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:29.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:54:29.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:54:29.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:54:29.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:54:29.779 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:54:29.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:54:29.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:29.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:54:29.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:29.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:30.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:31.077 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:31.070+0000 7fd8b02917c0 -1 Falling back to public interface 2024-09-17T18:54:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:54:31.108 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:54:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:31.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:32.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:32.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:32.441 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:54:32.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:54:32.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:32.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:32.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:33.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:33.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:33.775 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:54:33.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:54:33.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:33.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:33.814 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:33.810+0000 7fd8b02917c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:54:34.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:35.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:35.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:35.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:54:35.135 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:54:35.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:35.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:35.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:36.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:36.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:36.525 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:54:36.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:54:36.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:36.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:36.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:37.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:37.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:37.862 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:54:37.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:54:37.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:37.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:38.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:39.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:39.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:39.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:54:39.200 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:54:39.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:39.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:39.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:40.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:40.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:40.566 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:54:40.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:54:40.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:40.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:40.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:41.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:41.893 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:54:41.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:54:41.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:41.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:42.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:43.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:43.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:43.229 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:54:43.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:54:43.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:43.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:43.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:44.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:44.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:44.570 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:54:44.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:54:44.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:44.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:44.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:45.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:45.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:45.902 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:54:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:54:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:45.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:46.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:47.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:47.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:47.232 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:54:47.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:54:47.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:47.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:47.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:48.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:48.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:48.559 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:54:48.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:54:48.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:48.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:49.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:50.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:50.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:50.134 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:54:50.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:54:50.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:50.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:50.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:51.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:51.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:51.600 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:54:51.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:54:51.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:51.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:52.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:53.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:53.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:53.447 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:54:53.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:54:53.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:53.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:54.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:55.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:55.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:55.012 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:54:55.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:54:55.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:55.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:55.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:54:56.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:54:56.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:54:56.852 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:54:56.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:54:56.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:54:56.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:54:57.193 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2029082520,v1:127.0.0.1:6803/2029082520] [v2:127.0.0.1:6804/2029082520,v1:127.0.0.1:6805/2029082520] exists,up 5a73f560-d0f9-4229-b80c-05ebcaab8714 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T18:54:57.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:54:57.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:54:57.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:54:57.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:54:57.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:54:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:54:57.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3171ff2a-b632-4196-b7f6-9cc99af1c196' 2024-09-17T18:54:57.202 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:54:57.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:54:57.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCB0OlmzD3tDBAAMzLDtT+wYPyoyPgvowFeZw== 2024-09-17T18:54:57.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCB0OlmzD3tDBAAMzLDtT+wYPyoyPgvowFeZw=="}' 2024-09-17T18:54:57.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3171ff2a-b632-4196-b7f6-9cc99af1c196 -i td/osd-scrub-repair/1/new.json 2024-09-17T18:54:57.595 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:54:57.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T18:54:57.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCB0OlmzD3tDBAAMzLDtT+wYPyoyPgvowFeZw== --osd-uuid 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:54:57.648 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:57.642+0000 7fa3ffd9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:57.679 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:57.674+0000 7fa3ffd9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:57.681 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:57.674+0000 7fa3ffd9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:54:57.681 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:57.674+0000 7fa3ffd9e7c0 -1 bdev(0x55d6b2ccdc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T18:54:57.681 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:54:57.674+0000 7fa3ffd9e7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T18:55:03.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T18:55:03.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:55:03.792 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T18:55:03.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T18:55:03.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:55:04.207 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T18:55:04.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T18:55:04.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:55:04.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:55:04.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:55:04.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:55:04.240 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:04.238+0000 7f8bdaf057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:04.266 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:04.266+0000 7f8bdaf057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:04.267 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:04.266+0000 7f8bdaf057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:04.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T18:55:04.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:55:04.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:04.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:04.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:05.796 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:05.794+0000 7f8bdaf057c0 -1 Falling back to public interface 2024-09-17T18:55:05.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:05.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:05.898 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:55:05.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:55:05.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:05.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:06.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:07.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:07.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:07.221 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:55:07.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:55:07.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:07.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:07.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:08.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:08.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:08.582 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:55:08.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:55:08.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:08.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:08.761 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:08.758+0000 7f8bdaf057c0 -1 osd.1 0 log_to_monitors true 2024-09-17T18:55:08.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:09.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:09.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:09.903 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:55:09.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:55:09.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:09.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:10.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:11.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:11.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:11.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:55:11.242 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:55:11.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:11.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:11.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:12.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:12.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:12.595 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:55:12.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:55:12.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:12.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:12.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:13.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:13.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:13.924 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:55:13.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:55:13.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:13.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:14.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:15.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:15.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:15.252 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:55:15.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:55:15.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:15.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:15.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:16.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:16.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:16.599 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:55:16.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:55:16.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:16.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:16.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:17.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:17.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:17.933 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:55:17.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:55:17.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:17.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:18.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:19.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:19.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:19.259 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:55:19.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:55:19.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:19.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:19.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:20.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:20.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:20.653 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:55:20.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:55:20.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:20.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:20.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:21.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:21.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:21.985 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:55:21.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:55:21.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:21.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:22.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:23.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:23.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:23.594 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:55:23.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:55:23.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:23.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:24.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:25.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:25.493 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:55:25.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:55:25.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:25.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:26.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:27.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:27.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:27.234 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:55:27.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:55:27.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:27.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:27.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:28.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:28.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:28.962 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:55:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:55:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:29.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:30.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:30.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:30.374 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:55:30.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:55:30.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:30.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:31.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:32.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:32.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:32.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:55:32.098 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:55:32.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:32.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:55:32.451 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/992006705,v1:127.0.0.1:6811/992006705] [v2:127.0.0.1:6812/992006705,v1:127.0.0.1:6813/992006705] exists,up 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:55:32.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:55:32.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T18:55:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:55:32.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:55:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:55:32.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:55:32.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:55:32.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:55:32.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:55:32.461 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:55:32.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a56477c-3cf5-4649-9318-62941323bede 2024-09-17T18:55:32.462 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 7a56477c-3cf5-4649-9318-62941323bede 2024-09-17T18:55:32.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7a56477c-3cf5-4649-9318-62941323bede' 2024-09-17T18:55:32.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:55:32.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCk0OlmASWmHBAAUhSWpszbZXRqxGb3KT0QfQ== 2024-09-17T18:55:32.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCk0OlmASWmHBAAUhSWpszbZXRqxGb3KT0QfQ=="}' 2024-09-17T18:55:32.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a56477c-3cf5-4649-9318-62941323bede -i td/osd-scrub-repair/2/new.json 2024-09-17T18:55:32.868 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:55:32.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T18:55:32.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCk0OlmASWmHBAAUhSWpszbZXRqxGb3KT0QfQ== --osd-uuid 7a56477c-3cf5-4649-9318-62941323bede 2024-09-17T18:55:32.924 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:32.921+0000 7fa75b2dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:32.969 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:32.965+0000 7fa75b2dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:32.969+0000 7fa75b2dd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:32.969+0000 7fa75b2dd7c0 -1 bdev(0x56360918bc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T18:55:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:32.969+0000 7fa75b2dd7c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T18:55:39.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T18:55:39.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:55:39.197 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T18:55:39.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T18:55:39.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:55:39.602 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T18:55:39.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T18:55:39.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:55:39.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:55:39.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:55:39.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:55:39.636 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:39.633+0000 7f5be2df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:39.639 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:39.637+0000 7f5be2df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:39.640 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:39.637+0000 7f5be2df47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:55:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T18:55:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:55:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:55:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:55:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:55:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:39.927 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:55:39.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:55:39.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:39.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:40.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:40.932 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:40.929+0000 7f5be2df47c0 -1 Falling back to public interface 2024-09-17T18:55:41.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:41.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:41.276 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:55:41.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:55:41.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:41.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:41.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:42.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:42.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:42.612 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:55:42.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:55:42.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:42.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:42.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:43.838 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:55:43.837+0000 7f5be2df47c0 -1 osd.2 0 log_to_monitors true 2024-09-17T18:55:43.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:43.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:43.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:55:43.937 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:55:43.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:43.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:44.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:45.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:45.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:45.341 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:55:45.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:55:45.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:45.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:45.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:46.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:46.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:46.679 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:55:46.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:55:46.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:46.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:47.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:48.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:48.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:48.012 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:55:48.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:55:48.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:48.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:48.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:49.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:49.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:49.354 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:55:49.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:55:49.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:49.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:49.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:50.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:50.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:50.689 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:55:50.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:55:50.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:50.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:51.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:52.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:52.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:52.019 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:55:52.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:55:52.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:52.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:52.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:53.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:53.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:53.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:55:53.343 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:55:53.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:53.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:53.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:54.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:54.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:54.953 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:55:54.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:55:54.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:54.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:55.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:56.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:56.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:56.288 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:55:56.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:55:56.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:56.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:56.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:57.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:57.998 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:55:57.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:55:57.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:57.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:55:58.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:55:59.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:55:59.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:55:59.779 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:55:59.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:55:59.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:55:59.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:56:00.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:01.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:01.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:01.387 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:56:01.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:56:01.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:01.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:56:02.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:03.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:03.052 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:56:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:56:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:56:03.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:04.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:04.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:04.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:56:04.742 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:56:04.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:04.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:56:05.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:06.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:06.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:06.523 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:56:06.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:56:06.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:06.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:56:06.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:07.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:07.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:07.984 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:56:07.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:56:07.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:07.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:56:08.329 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1702226784,v1:127.0.0.1:6819/1702226784] [v2:127.0.0.1:6820/1702226784,v1:127.0.0.1:6821/1702226784] exists,up 7a56477c-3cf5-4649-9318-62941323bede 2024-09-17T18:56:08.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:56:08.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:56:08.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:56:08.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-09-17T18:56:08.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:56:08.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:56:08.336 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:56:08.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:56:08.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:56:08.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:56:08.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:56:08.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:56:08.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:56:08.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:56:08.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:56:08.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 f3d2ae3f-920b-4385-94d1-56ff789a7a94' 2024-09-17T18:56:08.339 INFO:tasks.workunit.client.0.smithi105.stdout:add osd3 f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:56:08.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:56:08.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDI0OlmERtAFRAA6WWJK3sAC/eOsaniqTBcVg== 2024-09-17T18:56:08.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDI0OlmERtAFRAA6WWJK3sAC/eOsaniqTBcVg=="}' 2024-09-17T18:56:08.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f3d2ae3f-920b-4385-94d1-56ff789a7a94 -i td/osd-scrub-repair/3/new.json 2024-09-17T18:56:08.742 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:56:08.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-17T18:56:08.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDI0OlmERtAFRAA6WWJK3sAC/eOsaniqTBcVg== --osd-uuid f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:56:08.799 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:08.796+0000 7fd6e27c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:56:08.827 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:08.824+0000 7fd6e27c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:56:08.828 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:08.824+0000 7fd6e27c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:56:08.829 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:08.824+0000 7fd6e27c27c0 -1 bdev(0x56299ad9bc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-17T18:56:08.829 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:08.824+0000 7fd6e27c27c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-17T18:56:14.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-17T18:56:14.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:56:14.964 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd3 key to auth repository 2024-09-17T18:56:14.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T18:56:14.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:56:15.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T18:56:15.375 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.3 2024-09-17T18:56:15.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:56:15.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:56:15.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:56:15.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:56:15.407 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:15.404+0000 7f654ebd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:56:15.432 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:15.428+0000 7f654ebd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:56:15.433 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:15.428+0000 7f654ebd37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:56:15.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T18:56:15.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:56:15.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:56:15.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:56:15.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:56:15.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:15.720 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:56:15.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:56:15.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:15.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:16.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:16.969 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:16.964+0000 7f654ebd37c0 -1 Falling back to public interface 2024-09-17T18:56:17.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:17.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:17.024 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:56:17.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:56:17.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:17.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:17.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:18.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:18.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:18.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:56:18.359 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:56:18.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:18.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:18.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:19.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:19.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:19.693 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:56:19.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:56:19.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:19.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:19.785 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:56:19.780+0000 7f654ebd37c0 -1 osd.3 0 log_to_monitors true 2024-09-17T18:56:20.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:21.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:21.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:21.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:56:21.052 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:56:21.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:21.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:21.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:22.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:22.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:22.474 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:56:22.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:56:22.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:22.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:22.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:23.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:23.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:23.962 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:56:23.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:56:23.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:23.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:24.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:25.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:25.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:25.300 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:56:25.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:56:25.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:25.300 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:25.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:26.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:26.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:26.704 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:56:26.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:56:26.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:26.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:27.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:28.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:28.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:28.040 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:56:28.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:56:28.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:28.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:28.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:29.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:29.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:29.377 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:56:29.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:56:29.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:29.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:29.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:30.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:30.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:30.715 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:56:30.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:56:30.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:30.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:31.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:32.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:32.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:32.042 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:56:32.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:56:32.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:32.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:32.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:33.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:33.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:33.374 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:56:33.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:56:33.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:33.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:34.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:35.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:35.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:35.023 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:56:35.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:56:35.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:35.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:35.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:36.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:36.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:36.909 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:56:36.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:56:36.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:36.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:37.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:38.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:38.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:38.345 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:56:38.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:56:38.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:38.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:38.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:39.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:39.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:39.939 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:56:39.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:56:39.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:39.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:40.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:41.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:41.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:41.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:56:41.594 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:56:41.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:41.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:41.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:42.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:42.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:42.989 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:56:42.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:56:42.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:42.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:43.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:56:44.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:56:44.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:56:44.484 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T18:56:44.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T18:56:44.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:56:44.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:56:44.821 INFO:tasks.workunit.client.0.smithi105.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3768574711,v1:127.0.0.1:6827/3768574711] [v2:127.0.0.1:6828/3768574711,v1:127.0.0.1:6829/3768574711] exists,up f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:56:44.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:56:44.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:56:44.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:56:44.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-09-17T18:56:44.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T18:56:45.250 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T18:56:45.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T18:56:45.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T18:56:45.736 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T18:56:45.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:56:46.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T18:56:49.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-09-17T18:56:49.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:56:49.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:56:49.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:56:49.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:56:49.061 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:56:49.061 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:56:49.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:56:49.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:56:49.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:56:49.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:56:49.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:56:49.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:56:49.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:56:49.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:56:49.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:56:49.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:56:49.545 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:56:49.545 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:56:49.545 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:56:49.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:56:49.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:49.546 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:56:49.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-17T18:56:49.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-17T18:56:49.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-17T18:56:49.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:49.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:56:49.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672984 2024-09-17T18:56:49.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672984 2024-09-17T18:56:49.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672984' 2024-09-17T18:56:49.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:49.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:56:50.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T18:56:50.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T18:56:50.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672984 2-64424509455' 2024-09-17T18:56:50.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:50.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:56:50.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345924 2024-09-17T18:56:50.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345924 2024-09-17T18:56:50.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672984 2-64424509455 3-85899345924' 2024-09-17T18:56:50.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:50.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-17T18:56:50.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:50.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:56:50.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-17T18:56:50.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:50.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-17T18:56:50.152 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836512 2024-09-17T18:56:50.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-17T18:56:50.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:56:50.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836512 2024-09-17T18:56:50.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:56:51.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:56:51.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:56:51.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-17T18:56:51.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:51.776 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672984 2024-09-17T18:56:51.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:51.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:56:51.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672984 2024-09-17T18:56:51.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:51.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672984 2024-09-17T18:56:51.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-17T18:56:51.782 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672984 2024-09-17T18:56:51.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:56:52.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-17T18:56:52.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:52.130 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-17T18:56:52.130 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:52.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:56:52.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-17T18:56:52.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:52.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T18:56:52.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-17T18:56:52.135 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509455 2024-09-17T18:56:52.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:56:52.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-17T18:56:52.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:52.476 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345924 2024-09-17T18:56:52.477 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:52.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:56:52.479 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345924 2024-09-17T18:56:52.479 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:52.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345924 2024-09-17T18:56:52.481 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345924 2024-09-17T18:56:52.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345924' 2024-09-17T18:56:52.482 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:56:52.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345924 -lt 85899345924 2024-09-17T18:56:52.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:56:52.880 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:56:52.880 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:56:53.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T18:56:53.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:56:53.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:56:53.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:56:53.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:56:53.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:56:53.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:56:53.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:56:53.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T18:56:53.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:56:53.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:56:53.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-17T18:56:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T18:56:53.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-09-17T18:56:54.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T18:56:54.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T18:56:55.237 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T18:56:55.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T18:56:56.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-17T18:56:56.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-17T18:56:56.846 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 2 allow_ec_overwrites to true 2024-09-17T18:56:56.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T18:56:56.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:56:56.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:56:56.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:56:56.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:56:56.869 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:56:56.869 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:56:56.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:56:56.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:56:56.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:56:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:56:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:56:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:56:56.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:56:56.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:56:56.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:56:57.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:56:57.332 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:56:57.332 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:56:57.333 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:56:57.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:56:57.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:57.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:56:57.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836515 2024-09-17T18:56:57.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836515 2024-09-17T18:56:57.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515' 2024-09-17T18:56:57.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:57.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:56:57.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672986 2024-09-17T18:56:57.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672986 2024-09-17T18:56:57.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672986' 2024-09-17T18:56:57.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:57.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:56:57.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-17T18:56:57.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-17T18:56:57.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509458' 2024-09-17T18:56:57.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:56:57.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:56:57.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345926 2024-09-17T18:56:57.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345926 2024-09-17T18:56:57.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836515 1-42949672986 2-64424509458 3-85899345926' 2024-09-17T18:56:57.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:57.906 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836515 2024-09-17T18:56:57.906 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:57.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:56:57.907 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836515 2024-09-17T18:56:57.907 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836515 2024-09-17T18:56:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-17T18:56:57.910 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836515 2024-09-17T18:56:57.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:56:58.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836515 2024-09-17T18:56:58.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:56:59.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:56:59.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:56:59.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-17T18:56:59.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:59.606 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672986 2024-09-17T18:56:59.607 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:59.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:56:59.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672986 2024-09-17T18:56:59.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:59.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672986 2024-09-17T18:56:59.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-09-17T18:56:59.611 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672986 2024-09-17T18:56:59.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:56:59.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672987 -lt 42949672986 2024-09-17T18:56:59.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:56:59.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-17T18:56:59.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:56:59.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:56:59.971 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-17T18:56:59.971 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:56:59.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-17T18:56:59.973 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509458 2024-09-17T18:56:59.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-17T18:56:59.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:57:00.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-17T18:57:00.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:00.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345926 2024-09-17T18:57:00.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:00.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:57:00.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345926 2024-09-17T18:57:00.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:00.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345926 2024-09-17T18:57:00.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-17T18:57:00.345 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345926 2024-09-17T18:57:00.346 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:57:00.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345926 -lt 85899345926 2024-09-17T18:57:00.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:57:00.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:00.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:01.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:57:01.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:57:01.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:57:01.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:57:01.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:57:01.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:57:01.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:57:01.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:57:01.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:57:01.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:57:01.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:01.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:01.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:57:01.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:57:01.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:57:01.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T18:57:01.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T18:57:01.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T18:57:02.279 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T18:57:02.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T18:57:02.807 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T18:57:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T18:57:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T18:57:02.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T18:57:03.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-17T18:57:03.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T18:57:03.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T18:57:03.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:57:03.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T18:57:03.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-17T18:57:03.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-17T18:57:03.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-17T18:57:03.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T18:57:03.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T18:57:03.593 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T18:57:03.593 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:57:03.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T18:57:03.928 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:03.928 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:03.928 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T18:57:03.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2024-09-17T18:57:03.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-09-17T18:57:03.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-17T18:57:03.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-09-17T18:57:03.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:57:03.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:57:03.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:57:03.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:57:03.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:57:03.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:57:03.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:57:03.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:57:03.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:57:03.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:57:03.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:57:04.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:57:04.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:57:04.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:57:04.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:57:04.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:57:04.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:57:04.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:57:04.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T18:57:06.585 INFO:tasks.workunit.client.0.smithi105.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T18:57:07.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:57:07.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:57:07.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:57:07.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:57:07.129 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:57:07.129 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:57:07.129 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:57:07.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:57:07.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:57:07.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:57:07.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:57:07.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:57:07.131 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T18:57:07.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:57:07.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:57:07.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:57:07.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:57:07.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:57:07.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:57:07.166 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:07.163+0000 7f556f6b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:07.169 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:07.163+0000 7f556f6b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:07.171 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:07.167+0000 7f556f6b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:07.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:57:07.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:57:07.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:57:07.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:57:07.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:57:07.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:07.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:57:07.465 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:57:07.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:07.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:07.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:08.473 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:08.467+0000 7f556f6b17c0 -1 Falling back to public interface 2024-09-17T18:57:08.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:08.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:08.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:57:08.795 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:08.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:08.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:09.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:10.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:10.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:57:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:10.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:11.285 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:11.279+0000 7f556f6b17c0 -1 osd.3 38 log_to_monitors true 2024-09-17T18:57:11.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:11.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:11.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:57:11.458 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:57:11.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:11.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:11.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:12.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:12.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:12.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:57:12.796 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:57:12.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:12.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:13.192 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [20,38) [v2:127.0.0.1:6826/3453054762,v1:127.0.0.1:6827/3453054762] [v2:127.0.0.1:6828/3453054762,v1:127.0.0.1:6829/3453054762] exists,up f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:57:13.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:57:13.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:57:13.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:57:13.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:57:13.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:57:13.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:57:13.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:57:13.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:57:13.194 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:57:13.194 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:57:13.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:57:13.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:57:13.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:57:13.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:57:13.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:57:13.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:57:13.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:57:13.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:57:13.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:57:13.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:57:13.634 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:13.634 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:13.634 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:57:13.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:57:13.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:13.635 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:57:13.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836519 2024-09-17T18:57:13.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836519 2024-09-17T18:57:13.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519' 2024-09-17T18:57:13.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:13.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:57:13.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672991 2024-09-17T18:57:13.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672991 2024-09-17T18:57:13.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672991' 2024-09-17T18:57:13.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:13.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:57:14.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509462 2024-09-17T18:57:14.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509462 2024-09-17T18:57:14.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672991 2-64424509462' 2024-09-17T18:57:14.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:14.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:57:14.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-17T18:57:14.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-17T18:57:14.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836519 1-42949672991 2-64424509462 3-184683593730' 2024-09-17T18:57:14.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:14.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836519 2024-09-17T18:57:14.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:14.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:57:14.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836519 2024-09-17T18:57:14.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:14.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836519 2024-09-17T18:57:14.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-09-17T18:57:14.216 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836519 2024-09-17T18:57:14.216 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:57:14.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836518 -lt 21474836519 2024-09-17T18:57:14.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:57:15.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:57:15.531 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:57:15.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836519 -lt 21474836519 2024-09-17T18:57:15.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:15.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:15.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672991 2024-09-17T18:57:15.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:57:15.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672991 2024-09-17T18:57:15.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:15.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672991 2024-09-17T18:57:15.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672991' 2024-09-17T18:57:15.866 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672991 2024-09-17T18:57:15.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:57:16.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672991 -lt 42949672991 2024-09-17T18:57:16.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:16.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:16.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509462 2024-09-17T18:57:16.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:57:16.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509462 2024-09-17T18:57:16.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:16.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509462 2024-09-17T18:57:16.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-09-17T18:57:16.228 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509462 2024-09-17T18:57:16.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:57:16.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509462 2024-09-17T18:57:16.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:16.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-184683593730 2024-09-17T18:57:16.576 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:16.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:57:16.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-184683593730 2024-09-17T18:57:16.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:16.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-17T18:57:16.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-09-17T18:57:16.580 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 184683593730 2024-09-17T18:57:16.580 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:57:16.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-17T18:57:16.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:57:16.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:16.920 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:17.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:57:17.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:57:17.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:57:17.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:57:17.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:57:17.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:57:17.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:57:17.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:57:17.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:57:17.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:57:17.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:17.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:18.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:57:18.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:57:18.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:57:18.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T18:57:18.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:57:18.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:57:18.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:57:18.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:57:18.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:18.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:57:19.022 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:57:19.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:19.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:57:19.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:19.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:57:19.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:57:19.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:19.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:19.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:19.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:19.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:19.039 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:19.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:56:55.144550+0000 '>' 2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:19.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:20.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:20.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:20.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:20.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:20.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:20.391 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:20.391 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:20.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:56:55.144550+0000 '>' 2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:20.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:21.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:21.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:21.724 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:21.724 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:21.724 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:21.724 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:21.725 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:22.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:56:55.144550+0000 '>' 2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:22.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:23.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:23.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:23.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:23.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:23.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:23.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:23.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:23.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:19.365158+0000 '>' 2024-09-17T18:56:55.144550+0000 2024-09-17T18:57:23.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:57:23.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:57:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:57:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:57:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:57:25.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:57:25.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:57:25.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:57:25.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:57:25.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:57:25.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T18:57:26.330 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T18:57:26.330 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:57:26.330 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T18:57:26.330 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:57:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:57:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:57:26.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:57:26.625 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T18:57:26.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:57:26.626 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:57:26.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:57:26.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:57:26.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:57:26.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:57:26.660 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:26.655+0000 7f6553ce17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:26.688 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:26.683+0000 7f6553ce17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:26.689 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:26.687+0000 7f6553ce17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:26.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:57:26.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:57:26.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:57:26.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:57:26.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:57:26.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:26.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:57:26.961 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:57:26.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:26.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:27.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:28.173 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:28.167+0000 7f6553ce17c0 -1 Falling back to public interface 2024-09-17T18:57:28.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:28.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:28.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:57:28.322 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:28.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:28.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:28.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:29.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:29.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:29.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:57:29.646 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:29.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:29.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:29.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:30.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:30.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:30.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:57:30.963 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:57:30.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:30.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:31.145 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:31.139+0000 7f6553ce17c0 -1 osd.3 45 log_to_monitors true 2024-09-17T18:57:31.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:32.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:32.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:32.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:57:32.320 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:57:32.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:32.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/1008494951,v1:127.0.0.1:6827/1008494951] [v2:127.0.0.1:6828/1008494951,v1:127.0.0.1:6829/1008494951] exists,up f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:57:32.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:57:32.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:57:32.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:57:32.779 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:57:32.779 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:57:32.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:57:32.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:57:32.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:57:32.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:57:32.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:57:32.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:57:32.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:57:32.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:57:32.896 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:57:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:57:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:57:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:57:33.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:33.399 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:57:33.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-17T18:57:33.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-17T18:57:33.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-17T18:57:33.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:33.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:57:33.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672996 2024-09-17T18:57:33.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672996 2024-09-17T18:57:33.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672996' 2024-09-17T18:57:33.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:33.702 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:57:33.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509467 2024-09-17T18:57:33.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509467 2024-09-17T18:57:33.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509467' 2024-09-17T18:57:33.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:33.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:57:33.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364802 2024-09-17T18:57:33.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364802 2024-09-17T18:57:33.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509467 3-214748364802' 2024-09-17T18:57:33.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:33.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-17T18:57:33.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:33.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:57:33.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-17T18:57:33.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:33.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-17T18:57:33.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-17T18:57:33.994 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836524 2024-09-17T18:57:33.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:57:34.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836524 2024-09-17T18:57:34.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:57:35.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:57:35.312 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:57:35.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-17T18:57:35.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:35.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672996 2024-09-17T18:57:35.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:35.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:57:35.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672996 2024-09-17T18:57:35.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:35.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672996 2024-09-17T18:57:35.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-17T18:57:35.648 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672996 2024-09-17T18:57:35.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:57:35.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-17T18:57:35.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:35.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509467 2024-09-17T18:57:35.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:35.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:57:35.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509467 2024-09-17T18:57:35.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:35.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509467 2024-09-17T18:57:35.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-09-17T18:57:35.982 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509467 2024-09-17T18:57:35.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:57:36.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509467 -lt 64424509467 2024-09-17T18:57:36.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:36.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364802 2024-09-17T18:57:36.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:36.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:57:36.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364802 2024-09-17T18:57:36.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:36.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364802 2024-09-17T18:57:36.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-09-17T18:57:36.327 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 214748364802 2024-09-17T18:57:36.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:57:36.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364802 -lt 214748364802 2024-09-17T18:57:36.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:57:36.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:36.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:57:37.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:57:37.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:57:37.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:57:37.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:57:37.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:37.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:37.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:57:37.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:57:37.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:57:37.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:57:37.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:57:37.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-09-17T18:57:37.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T18:57:37.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T18:57:37.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-17T18:57:37.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:57:37.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:57:38.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:57:40.367 INFO:tasks.workunit.client.0.smithi105.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:57:40.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:57:40.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:57:40.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:57:40.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:57:40.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:57:40.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:57:40.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:57:40.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:57:40.913 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T18:57:40.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:57:40.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:57:40.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:57:40.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:57:40.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:57:40.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:57:40.948 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:40.943+0000 7fbf13deb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:40.951 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:40.947+0000 7fbf13deb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:40.952 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:40.947+0000 7fbf13deb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:41.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:57:41.249 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:57:41.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:57:41.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:41.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:42.196 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:42.191+0000 7fbf13deb7c0 -1 Falling back to public interface 2024-09-17T18:57:42.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:42.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:42.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:57:42.605 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:42.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:42.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:57:42.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:43.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:43.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:43.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:57:43.921 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:43.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:43.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:57:44.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:57:44.945 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:57:44.939+0000 7fbf13deb7c0 -1 osd.1 52 log_to_monitors true 2024-09-17T18:57:45.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:57:45.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:57:45.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:57:45.238 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:57:45.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:57:45.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 57 up_thru 46 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/1701849217,v1:127.0.0.1:6811/1701849217] [v2:127.0.0.1:6812/1701849217,v1:127.0.0.1:6813/1701849217] exists,up 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:57:45.680 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:57:45.681 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:57:45.681 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:57:45.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:57:45.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:57:45.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:57:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:57:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:57:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:57:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:57:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:57:45.795 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:57:46.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:57:46.220 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:57:46.220 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:57:46.220 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:57:46.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:57:46.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:46.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:57:46.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836527 2024-09-17T18:57:46.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836527 2024-09-17T18:57:46.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527' 2024-09-17T18:57:46.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:46.571 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:57:46.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-17T18:57:46.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-17T18:57:46.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-244813135874' 2024-09-17T18:57:46.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:46.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:57:47.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509470 2024-09-17T18:57:47.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509470 2024-09-17T18:57:47.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-244813135874 2-64424509470' 2024-09-17T18:57:47.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:57:47.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:57:47.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364806 2024-09-17T18:57:47.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364806 2024-09-17T18:57:47.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-244813135874 2-64424509470 3-214748364806' 2024-09-17T18:57:47.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:47.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836527 2024-09-17T18:57:47.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:47.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:57:47.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836527 2024-09-17T18:57:47.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:47.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836527 2024-09-17T18:57:47.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-09-17T18:57:47.361 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836527 2024-09-17T18:57:47.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:57:47.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836527 -lt 21474836527 2024-09-17T18:57:47.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:47.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-17T18:57:47.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:47.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:57:47.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-17T18:57:47.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:47.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-17T18:57:47.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-17T18:57:47.697 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 244813135874 2024-09-17T18:57:47.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:57:48.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-17T18:57:48.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:48.029 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509470 2024-09-17T18:57:48.029 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:48.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:57:48.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509470 2024-09-17T18:57:48.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:48.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509470 2024-09-17T18:57:48.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-17T18:57:48.033 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509470 2024-09-17T18:57:48.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:57:48.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-17T18:57:48.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:57:48.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364806 2024-09-17T18:57:48.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:57:48.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:57:48.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364806 2024-09-17T18:57:48.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:57:48.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364806 2024-09-17T18:57:48.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-09-17T18:57:48.371 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 214748364806 2024-09-17T18:57:48.371 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:57:48.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364805 -lt 214748364806 2024-09-17T18:57:48.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:57:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:57:49.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:57:50.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364806 -lt 214748364806 2024-09-17T18:57:50.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:57:50.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:50.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:50.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:57:50.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:57:50.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:57:50.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:57:50.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:57:50.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:57:50.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:57:50.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:57:50.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:57:50.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:57:50.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:57:50.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:57:51.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:57:51.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:57:51.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:57:51.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T18:57:51.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:57:51.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:57:51.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:57:51.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:57:51.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T18:57:51.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T18:57:51.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:57:51.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:57:51.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:51.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:51.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:51.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:51.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:51.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:57:52.165 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:57:52.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:52.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:57:52.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:52.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:57:52.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:57:52.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:52.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:52.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:52.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:52.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:52.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:52.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:19.365158+0000 '>' 2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:52.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:53.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:53.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:53.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:53.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:53.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:53.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:53.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:53.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:19.365158+0000 '>' 2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:53.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:54.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:54.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:54.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:54.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:54.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:54.818 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:54.818 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:55.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:19.365158+0000 '>' 2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:55.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:56.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:56.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:56.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:56.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:56.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:56.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:56.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:56.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:19.365158+0000 '>' 2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:56.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:57:57.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:57:57.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:57:57.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:57:57.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:57:57.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:57:57.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:57:57.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:57:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:52.235167+0000 '>' 2024-09-17T18:57:19.365158+0000 2024-09-17T18:57:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:57:57.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:57:59.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:57:59.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:57:59.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:57:59.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:57:59.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:57:59.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:57:59.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:00.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:58:00.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T18:58:00.742 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T18:58:00.742 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T18:58:00.743 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T18:58:00.743 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T18:58:01.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:01.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:01.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:01.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:01.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:01.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:01.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:01.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:58:01.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:58:01.040 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T18:58:01.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:58:01.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:58:01.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:58:01.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:01.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:01.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:01.074 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:58:01.070+0000 7f1c8664c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:01.077 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:58:01.070+0000 7f1c8664c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:01.078 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:58:01.074+0000 7f1c8664c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:01.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:58:01.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:01.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:58:01.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:01.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:01.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:01.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:01.405 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T18:58:01.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:01.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:01.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:02.376 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:58:02.370+0000 7f1c8664c7c0 -1 Falling back to public interface 2024-09-17T18:58:02.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:02.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:02.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:02.736 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:58:02.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:02.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:03.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:04.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:04.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:04.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:04.066 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:58:04.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:04.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:04.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:05.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:05.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:05.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:05.396 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T18:58:05.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:05.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:05.422 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:58:05.418+0000 7f1c8664c7c0 -1 osd.3 59 log_to_monitors true 2024-09-17T18:58:05.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:06.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:06.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:06.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:06.777 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T18:58:06.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:06.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/3475581045,v1:127.0.0.1:6827/3475581045] [v2:127.0.0.1:6828/3475581045,v1:127.0.0.1:6829/3475581045] exists,up f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:07.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:07.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:07.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:07.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:07.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:07.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:07.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:07.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:07.205 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:07.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:07.536 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T18:58:07.536 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T18:58:07.536 INFO:tasks.workunit.client.0.smithi105.stderr:3' 2024-09-17T18:58:07.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:07.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:07.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:07.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-17T18:58:07.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-17T18:58:07.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-17T18:58:07.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:07.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:07.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-17T18:58:07.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-17T18:58:07.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-244813135879' 2024-09-17T18:58:07.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:07.856 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:08.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509476 2024-09-17T18:58:08.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509476 2024-09-17T18:58:08.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-244813135879 2-64424509476' 2024-09-17T18:58:08.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:08.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:08.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939650 2024-09-17T18:58:08.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939650 2024-09-17T18:58:08.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-244813135879 2-64424509476 3-270582939650' 2024-09-17T18:58:08.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:08.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-17T18:58:08.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:08.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:08.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-17T18:58:08.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:08.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-17T18:58:08.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-17T18:58:08.142 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836533 2024-09-17T18:58:08.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:08.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836531 -lt 21474836533 2024-09-17T18:58:08.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:58:09.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:58:09.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:09.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836533 2024-09-17T18:58:09.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:09.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135879 2024-09-17T18:58:09.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:09.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:09.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135879 2024-09-17T18:58:09.786 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:09.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-17T18:58:09.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-09-17T18:58:09.787 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 244813135879 2024-09-17T18:58:09.788 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:58:10.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135879 -lt 244813135879 2024-09-17T18:58:10.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:10.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509476 2024-09-17T18:58:10.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:10.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:58:10.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509476 2024-09-17T18:58:10.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509476 2024-09-17T18:58:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-09-17T18:58:10.113 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509476 2024-09-17T18:58:10.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:58:10.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509476 -lt 64424509476 2024-09-17T18:58:10.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:10.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-270582939650 2024-09-17T18:58:10.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:10.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:58:10.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-270582939650 2024-09-17T18:58:10.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:10.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939650 2024-09-17T18:58:10.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 270582939650' 2024-09-17T18:58:10.449 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 270582939650 2024-09-17T18:58:10.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:58:10.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939650 -lt 270582939650 2024-09-17T18:58:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:58:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:10.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:11.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:58:11.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:11.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:11.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:11.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:11.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:58:11.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:11.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:11.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:58:11.528 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:11.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:11.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:11.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:58:11.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:58:11.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:58:11.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:58:11.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:58:11.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 167298"' 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 167298' 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 167300"' 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 167300' 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 167298 167300' 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 167298 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/167301: /' 2024-09-17T18:58:11.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:58:11.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/167303: /' 2024-09-17T18:58:15.167 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.167 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:58:15.167 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:58:15.167 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:58:15.167 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:15.168 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:15.169 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:15.172 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:15.172 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:15.172 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:58:15.172 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:15.173 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: start osd.1 2024-09-17T18:58:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:1167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.334 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:15.335 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:15.336 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:15.338 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:15.339 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:15.340 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:15.340 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:15.340 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:58:15.340 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:58:15.340 INFO:tasks.workunit.client.0.smithi105.stderr:167303: start osd.2 2024-09-17T18:58:15.340 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:58:19.953 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:58:19.953 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2024-09-17T18:58:15.202+0000 7f694fc597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2024-09-17T18:58:15.206+0000 7f694fc597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2024-09-17T18:58:15.206+0000 7f694fc597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:19.954 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 0 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2024-09-17T18:58:16.506+0000 7f694fc597c0 -1 Falling back to public interface 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 1 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:19.955 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2024-09-17T18:58:19.426+0000 7f694fc597c0 -1 osd.1 64 log_to_monitors true 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 3 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:19.956 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:20.025 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers67303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T18:58:20.025 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:58:20.025 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2024-09-17T18:58:15.362+0000 7fbb670c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2024-09-17T18:58:15.366+0000 7fbb670c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2024-09-17T18:58:15.366+0000 7fbb670c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 0 2024-09-17T18:58:20.026 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2024-09-17T18:58:16.690+0000 7fbb670c07c0 -1 Falling back to public interface 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 1 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:20.027 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 3 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:20.028 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2024-09-17T18:58:19.962+0000 7fbb670c07c0 -1 osd.2 64 log_to_monitors true 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 4 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [57,64) [v2:127.0.0.1:6810/2254728596,v1:127.0.0.1:6811/2254728596] [v2:127.0.0.1:6812/2254728596,v1:127.0.0.1:6813/2254728596] exists,up 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:58:21.966 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 1 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 2 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: 3' 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836536 2024-09-17T18:58:21.968 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836536 2024-09-17T18:58:21.969 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536' 2024-09-17T18:58:21.969 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:22.174 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:22.174 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:22.174 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:22.174 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:22.174 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 4 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: osd.2 up in weight 1 up_from 71 up_thru 53 down_at 65 last_clean_interval [15,64) [v2:127.0.0.1:6818/2070136213,v1:127.0.0.1:6819/2070136213] [v2:127.0.0.1:6820/2070136213,v1:127.0.0.1:6821/2070136213] exists,up 7a56477c-3cf5-4649-9318-62941323bede 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:22.175 INFO:tasks.workunit.client.0.smithi105.stderr:167303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 1 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 2 2024-09-17T18:58:22.176 INFO:tasks.workunit.client.0.smithi105.stderr:167303: 3' 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836537 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836537 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537' 2024-09-17T18:58:22.177 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776130 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776130 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-292057776130' 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678018 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678018 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-292057776130 2-304942678018' 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:24.101 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939654 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939654 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836536 1-292057776130 2-304942678018 3-270582939654' 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836536 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836536 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836536 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: waiting osd.0 seq 21474836536 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836536 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:58:24.102 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836536 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776130 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:24.103 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776130 2024-09-17T18:58:24.395 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:24.395 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776131 2024-09-17T18:58:24.395 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776131 2024-09-17T18:58:24.395 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-292057776131' 2024-09-17T18:58:24.395 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:24.395 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=304942678019 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 304942678019 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-292057776131 2-304942678019' 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939655 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939655 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836537 1-292057776131 2-304942678019 3-270582939655' 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836537 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836537 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:24.396 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836537 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: waiting osd.0 seq 21474836537 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836535 -lt 21474836537 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836538 -lt 21474836537 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776131 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776131 2024-09-17T18:58:24.397 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:25.647 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776130 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: waiting osd.1 seq 292057776130 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776131 -lt 292057776130 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-304942678018 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-304942678018 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678018 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-09-17T18:58:25.648 INFO:tasks.workunit.client.0.smithi105.stderr:167301: waiting osd.2 seq 304942678018 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678019 -lt 304942678018 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-270582939654 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-270582939654 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939654 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 270582939654' 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: waiting osd.3 seq 270582939654 2024-09-17T18:58:25.649 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939655 -lt 270582939654 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:25.650 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:25.758 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("suntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776131 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: waiting osd.1 seq 292057776131 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776131 -lt 292057776131 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-304942678019 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-304942678019 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=304942678019 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 304942678019' 2024-09-17T18:58:25.759 INFO:tasks.workunit.client.0.smithi105.stderr:167303: waiting osd.2 seq 304942678019 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 304942678019 -lt 304942678019 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-270582939655 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-270582939655 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939655 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 270582939655' 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: waiting osd.3 seq 270582939655 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939655 -lt 270582939655 2024-09-17T18:58:25.760 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:25.761 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:27.254 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:58:27.255 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:58:27.256 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T18:58:27.257 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:58:27.257 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:58:27.257 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:58:27.257 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:58:27.257 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:58:27.315 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //tale") | not)' 2024-09-17T18:58:27.315 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:58:27.316 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T18:58:27.317 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:58:27.318 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:58:27.318 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:58:27.318 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:58:27.318 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:58:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:58:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:58:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:58:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:167301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:167301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:58:28.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 167300 2024-09-17T18:58:28.556 INFO:tasks.workunit.client.0.smithi105.stderr:home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:58:28.556 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:58:28.556 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T18:58:28.556 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T18:58:28.556 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T18:58:28.556 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:167303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T18:58:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:58:28.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T18:58:28.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:58:28.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:58:28.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:58:28.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:58:28.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T18:58:28.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T18:58:28.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:58:28.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:58:28.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:58:28.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:58:28.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:58:28.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:58:29.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:57:52.235167+0000 2024-09-17T18:58:29.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:58:29.491 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:58:29.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:57:52.235167+0000 2024-09-17T18:58:29.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:58:29.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:57:52.235167+0000 2024-09-17T18:58:29.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:58:29.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:58:29.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:58:29.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:58:29.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:58:29.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:58:29.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:58:29.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:58:29.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:57:52.235167+0000 '>' 2024-09-17T18:57:52.235167+0000 2024-09-17T18:58:29.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:58:30.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:58:30.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:58:30.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:58:30.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:58:30.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:58:30.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:58:30.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:58:31.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:58:29.530120+0000 '>' 2024-09-17T18:57:52.235167+0000 2024-09-17T18:58:31.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:58:31.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:58:33.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T18:58:33.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:58:33.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:58:33.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:58:33.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 170388"' 2024-09-17T18:58:33.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 170388' 2024-09-17T18:58:33.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 170391"' 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 170391' 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/170390: /' 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 170388 170391' 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 170388 2024-09-17T18:58:33.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:58:33.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/170393: /' 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:58:34.332 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-09-17T18:58:34.333 INFO:tasks.workunit.client.0.smithi105.stderr:170393: _ 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: hinfo_key 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: snapset 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:34.334 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:34.337 INFO:tasks.workunit.client.0.smithi105.stderr:170393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T18:58:34.339 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T18:58:34.339 INFO:tasks.workunit.client.0.smithi105.stderr:170393: start osd.2 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.374 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:58:34.375 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: _ 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: hinfo_key 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: snapset 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:58:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:58:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:170390: start osd.1 2024-09-17T18:58:39.367 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:58:39.367 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T18:58:39.367 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T18:58:39.367 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:39.367 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:39.367 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2024-09-17T18:58:34.382+0000 7fe28a9047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2024-09-17T18:58:34.406+0000 7fe28a9047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2024-09-17T18:58:34.406+0000 7fe28a9047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 0 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.368 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2024-09-17T18:58:35.686+0000 7fe28a9047c0 -1 Falling back to public interface 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 1 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2 2024-09-17T18:58:39.369 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 3 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:39.370 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2024-09-17T18:58:39.366+0000 7fe2very_ops 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2024-09-17T18:58:34.418+0000 7fd46ed0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2024-09-17T18:58:34.422+0000 7fd46ed0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2024-09-17T18:58:34.422+0000 7fd46ed0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:39.608 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 0 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2024-09-17T18:58:35.470+0000 7fd46ed0f7c0 -1 Falling back to public interface 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:39.609 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 1 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:39.610 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 3 2024-09-17T18:58:39.611 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:39.611 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:39.611 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2024-09-17T18:58:39.606+0000 7fd48a9047c0 -1 osd.2 73 log_to_monitors true 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 4 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: osd.2 up in weight 1 up_from 79 up_thru 53 down_at 74 last_clean_interval [71,73) [v2:127.0.0.1:6818/84809574,v1:127.0.0.1:6819/84809574] [v2:127.0.0.1:6820/84809574,v1:127.0.0.1:6821/84809574] exists,up 7a56477c-3cf5-4649-9318-62941323bede 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:41.622 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:41.623 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 1 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 2 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: 3' 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:41.624 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836542 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836542 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542' 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-he6ed0f7c0 -1 osd.1 73 log_to_monitors true 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 4 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: osd.1 up in weight 1 up_from 80 up_thru 71 down_at 74 last_clean_interval [68,73) [v2:127.0.0.1:6810/1867555509,v1:127.0.0.1:6811/1867555509] [v2:127.0.0.1:6812/1867555509,v1:127.0.0.1:6813/1867555509] exists,up 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:41.625 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:41.626 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 1 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 2 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: 3' 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:41.627 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836543 2024-09-17T18:58:41.628 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836543 2024-09-17T18:58:41.628 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543' 2024-09-17T18:58:41.628 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:43.741 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:43.742 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383682 2024-09-17T18:58:43.742 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383682 2024-09-17T18:58:43.742 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-343597383682' 2024-09-17T18:58:43.742 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:43.742 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416387 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416387 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-343597383682 2-339302416387' 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939661 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939661 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836543 1-343597383682 2-339302416387 3-270582939661' 2024-09-17T18:58:43.743 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836543 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836543 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836543 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-09-17T18:58:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:170390: waiting osd.0 seq 21474836543 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836541 -lt 21474836543 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836543 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:43.745 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-343597383682 2024-09-17T18:58:43.746 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:43.746 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:43.746 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-343597383682 2024-09-17T18:58:43.746 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170390lpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383683 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383683 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-343597383683' 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416386 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416386 2024-09-17T18:58:43.761 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-343597383683 2-339302416386' 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=270582939660 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 270582939660 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-343597383683 2-339302416386 3-270582939660' 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836542 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:43.762 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836542 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836542 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: waiting osd.0 seq 21474836542 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836541 -lt 21474836542 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:43.763 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836544 -lt 21474836542 2024-09-17T18:58:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-343597383683 2024-09-17T18:58:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-343597383683 2024-09-17T18:58:43.764 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:45.166 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383683 2024-09-17T18:58:45.166 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2024-09-17T18:58:45.166 INFO:tasks.workunit.client.0.smithi105.stderr:170393: waiting osd.1 seq 343597383683 2024-09-17T18:58:45.166 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383683 -lt 343597383683 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-339302416386 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-339302416386 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416386 2024-09-17T18:58:45.167 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 339302416386' 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: waiting osd.2 seq 339302416386 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416387 -lt 339302416386 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-270582939660 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-270582939660 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:45.168 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939660 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 270582939660' 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: waiting osd.3 seq 270582939660 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939661 -lt 270582939660 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:58:45.169 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:45.170 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains(: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383682 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: waiting osd.1 seq 343597383682 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383683 -lt 343597383682 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-339302416387 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-339302416387 2024-09-17T18:58:45.171 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416387 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 339302416387' 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: waiting osd.2 seq 339302416387 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416387 -lt 339302416387 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-270582939661 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:45.172 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-270582939661 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=270582939661 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 270582939661' 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: waiting osd.3 seq 270582939661 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 270582939661 -lt 270582939661 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:58:45.173 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:58:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:58:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:58:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:58:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(c"stale") | not)' 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:45.899 INFO:tasks.workunit.client.0.smithi105.stderr:170393: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:45.900 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:58:45.900 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:58:45.900 INFO:tasks.workunit.client.0.smithi105.stderr:170393: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:58:45.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:58:45.921 INFO:tasks.workunit.client.0.smithi105.stderr:ontains("stale") | not)' 2024-09-17T18:58:45.921 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:58:45.921 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:58:45.921 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:170390: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:170390: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 170391 2024-09-17T18:58:45.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:58:45.923 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:58:45.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:58:45.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T18:58:45.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:58:45.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:58:45.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:58:45.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-09-17T18:58:45.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T18:58:45.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T18:58:45.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 172769"' 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 172769' 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:58:45.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 172771"' 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 172771' 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 172769 172771' 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:58:45.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 172769 2024-09-17T18:58:45.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/172772: /' 2024-09-17T18:58:45.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:45.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/172774: /' 2024-09-17T18:58:49.163 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.164 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:58:49.164 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:58:49.164 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:58:49.164 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:58:49.164 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.164 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:58:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.167 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:49.167 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:58:49.167 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:49.167 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:58:49.167 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:49.167 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:49.168 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:49.168 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:49.168 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:49.169 INFO:tasks.workunit.client.0.smithi105.stderr:172772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:49.171 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:49.171 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:49.171 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:58:49.171 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:58:49.171 INFO:tasks.workunit.client.0.smithi105.stderr:172772: start osd.3 2024-09-17T18:58:49.171 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:58:49.689 INFO:tasks.workunit.client.0.smithi105.stderr:1172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:58:49.690 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:58:49.691 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T18:58:49.692 INFO:tasks.workunit.client.0.smithi105.stderr:172774: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:58:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:58:49.694 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:58:49.694 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:58:49.694 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:58:49.694 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:58:49.695 INFO:tasks.workunit.client.0.smithi105.stderr:172774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:58:49.696 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:58:49.697 INFO:tasks.workunit.client.0.smithi105.stderr:172774: start osd.1 2024-09-17T18:58:49.698 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:58:54.017 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:58:54.017 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:58:54.017 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2024-09-17T18:58:49.201+0000 7f071c7db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2024-09-17T18:58:49.201+0000 7f071c7db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2024-09-17T18:58:49.205+0000 7f071c7db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 0 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2024-09-17T18:58:50.733+0000 7f071c7db7c0 -1 Falling back to public interface 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 1 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:54.020 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 3 2024-09-17T18:58:54.021 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.021 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:54.021 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2024-09-17T18:58:53.549+0000 7f071c7db7c0 -1 osd.3 81 log_to_monitors true 2024-09-17T18:58:54.515 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers72774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:58:54.515 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:58:54.515 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:58:54.515 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2024-09-17T18:58:49.721+0000 7fb6a52337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2024-09-17T18:58:49.725+0000 7fb6a52337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2024-09-17T18:58:49.725+0000 7fb6a52337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:58:54.516 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 0 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2024-09-17T18:58:50.985+0000 7fb6a52337c0 -1 Falling back to public interface 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 1 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:54.517 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 3 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:54.518 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:56.300 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2024-09-17T18:58:54.513+0000 7fb6a52337c0 -.sh:976: wait_for_osd: sleep 1 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 4 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: osd.3 up in weight 1 up_from 86 up_thru 86 down_at 82 last_clean_interval [63,81) [v2:127.0.0.1:6810/1076019999,v1:127.0.0.1:6811/1076019999] [v2:127.0.0.1:6812/1076019999,v1:127.0.0.1:6813/1076019999] exists,up f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:56.301 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:56.302 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 1 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 2 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: 3' 2024-09-17T18:58:56.303 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:56.304 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:56.304 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:56.304 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836547 2024-09-17T18:58:56.304 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836547 2024-09-17T18:58:56.304 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547' 2024-09-17T18:58:56.304 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help1 osd.1 81 log_to_monitors true 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 4 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: osd.1 up in weight 1 up_from 89 up_thru 80 down_at 82 last_clean_interval [80,81) [v2:127.0.0.1:6826/2480435919,v1:127.0.0.1:6827/2480435919] [v2:127.0.0.1:6828/2480435919,v1:127.0.0.1:6829/2480435919] exists,up 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:58:56.580 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:58:56.581 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 1 2024-09-17T18:58:56.582 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 2 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: 3' 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836548 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836548 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548' 2024-09-17T18:58:56.583 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:57.896 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:57.896 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089347 2024-09-17T18:58:57.896 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089347 2024-09-17T18:58:57.896 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-382252089347' 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416392 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416392 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-382252089347 2-339302416392' 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187459 2024-09-17T18:58:57.897 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187459 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836548 1-382252089347 2-339302416392 3-369367187459' 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836548 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836548 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836548 2024-09-17T18:58:57.898 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: waiting osd.0 seq 21474836548 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836548 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089347 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089347 2024-09-17T18:58:57.899 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:57.900 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089347 2024-09-17T18:58:57.900 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089347' 2024-09-17T18:58:57.900 INFO:tasks.workunit.client.0.smithi105.stderr:172774: waiting osd.1 seq 382252089347 2024-09-17T18:58:57.900 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:58:58.672 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222ers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089346 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089346 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-382252089346' 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416391 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416391 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-382252089346 2-339302416391' 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:58:58.673 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=369367187458 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 369367187458 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836547 1-382252089346 2-339302416391 3-369367187458' 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836547 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836547 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836547 2024-09-17T18:58:58.674 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: waiting osd.0 seq 21474836547 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836546 -lt 21474836547 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836548 -lt 21474836547 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089346 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:58:58.675 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089346 2024-09-17T18:58:58.676 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089346 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: waiting osd.1 seq 382252089346 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089347 -lt 382252089346 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:00.218 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-339302416391 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-339302416391 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416391 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 339302416391' 2024-09-17T18:59:00.219 INFO:tasks.workunit.client.0.smithi105.stderr:172772: waiting osd.2 seq 339302416391 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416392 -lt 339302416391 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-369367187458 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-369367187458 2024-09-17T18:59:00.220 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187458 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 369367187458' 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: waiting osd.3 seq 369367187458 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187460 -lt 369367187458 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:59:00.221 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:00.222 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:00.222 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:59:00.222 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:00.222 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:00.222 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:00.222 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:00.296 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("s: flush_pg_stats: test 382252089347 -lt 382252089347 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-339302416392 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-339302416392 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416392 2024-09-17T18:59:00.297 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 339302416392' 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: waiting osd.2 seq 339302416392 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416392 -lt 339302416392 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-369367187459 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-369367187459 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=369367187459 2024-09-17T18:59:00.298 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: waiting osd.3 seq 369367187459 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187458 -lt 369367187459 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 369367187460 -lt 369367187459 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:00.299 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:00.300 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:59:00.300 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:00.300 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:00.300 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:00.300 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains(tale") | not)' 2024-09-17T18:59:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:59:01.749 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:01.750 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:59:01.751 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:59:01.751 INFO:tasks.workunit.client.0.smithi105.stderr:172772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:59:01.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:59:01.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:59:01.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 172771 2024-09-17T18:59:01.885 INFO:tasks.workunit.client.0.smithi105.stderr:"stale") | not)' 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:59:01.886 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:172774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T18:59:01.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:59:01.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T18:59:01.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T18:59:01.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T18:59:01.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T18:59:01.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T18:59:02.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T18:59:02.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T18:59:02.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T18:59:02.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T18:59:02.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:59:02.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:59:02.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:59:02.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:59:02.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T18:58:29.530120+0000 2024-09-17T18:59:02.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T18:59:02.849 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T18:59:02.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T18:58:29.530120+0000 2024-09-17T18:59:02.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T18:59:02.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T18:58:29.530120+0000 2024-09-17T18:59:02.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T18:59:02.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T18:59:02.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:59:02.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:59:02.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:59:02.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:59:02.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:59:02.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:59:03.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:58:29.530120+0000 '>' 2024-09-17T18:58:29.530120+0000 2024-09-17T18:59:03.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:59:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:59:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:59:04.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:59:04.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:59:04.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:59:04.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:59:04.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:59:04.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:58:29.530120+0000 '>' 2024-09-17T18:58:29.530120+0000 2024-09-17T18:59:04.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T18:59:05.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T18:59:05.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T18:59:05.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T18:59:05.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T18:59:05.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T18:59:05.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T18:59:05.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T18:59:05.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T18:59:03.670976+0000 '>' 2024-09-17T18:58:29.530120+0000 2024-09-17T18:59:05.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T18:59:05.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T18:59:07.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T18:59:07.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:59:07.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:59:07.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:59:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 175738"' 2024-09-17T18:59:07.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 175738' 2024-09-17T18:59:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 175740"' 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/175741: /' 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 175740' 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 175738 175740' 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 175738 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:59:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/175744: /' 2024-09-17T18:59:09.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:59:09.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: _ 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: hinfo_key 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: snapset 2024-09-17T18:59:09.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:59:09.063 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:59:09.066 INFO:tasks.workunit.client.0.smithi105.stderr:175744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:59:09.067 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T18:59:09.068 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T18:59:09.068 INFO:tasks.workunit.client.0.smithi105.stderr:175744: start osd.1 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.094 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T18:59:09.095 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: _ 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: hinfo_key 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: snapset 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T18:59:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:59:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:59:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:59:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:59:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T18:59:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T18:59:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:59:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T18:59:09.100 INFO:tasks.workunit.client.0.smithi105.stderr:175741: start osd.3 2024-09-17T18:59:13.823 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2024-09-17T18:59:09.129+0000 7f1db6df17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2024-09-17T18:59:09.129+0000 7f1db6df17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2024-09-17T18:59:09.133+0000 7f1db6df17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T18:59:13.824 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 0 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2024-09-17T18:59:10.441+0000 7f1db6df17c0 -1 Falling back to public interface 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 1 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:13.825 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 3 2024-09-17T18:59:13.826 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:13.827 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:59:14.367 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-17T18:59:14.367 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T18:59:14.367 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T18:59:14.367 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T18:59:14.367 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2024-09-17T18:59:09.109+0000 7f2edb6c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2024-09-17T18:59:09.113+0000 7f2edb6c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2024-09-17T18:59:09.113+0000 7f2edb6c67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 0 2024-09-17T18:59:14.368 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2024-09-17T18:59:10.617+0000 7f2edb6c67c0 -1 Falling back to public interface 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 1 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:59:14.369 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 3 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:59:14.370 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2024-09-17T18:59:14.365+0000 7f2edb6c67c0 -1 osd.1 91 log_to_monitors true 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 4 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:15.826 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: osd.1 up in weight 1 up_from 95 up_thru 89 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6826/1746367320,v1:127.0.0.1:6827/1746367320] [v2:127.0.0.1:6828/1746367320,v1:127.0.0.1:6829/1746367320] exists,up 3171ff2a-b632-4196-b7f6-9cc99af1c196 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:59:15.827 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 1 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 2 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: 3' 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836553 2024-09-17T18:59:15.828 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836553 2024-09-17T18:59:15.829 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553' 2024-09-17T18:59:15.829 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2024-09-17T18:59:14.381+0000 7f1db6df17c0 -1 osd.3 91 log_to_monitors true 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 4 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: osd.3 up in weight 1 up_from 95 up_thru 89 down_at 92 last_clean_interval [86,91) [v2:127.0.0.1:6810/1829956966,v1:127.0.0.1:6811/1829956966] [v2:127.0.0.1:6812/1829956966,v1:127.0.0.1:6813/1829956966] exists,up f3d2ae3f-920b-4385-94d1-56ff789a7a94 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T18:59:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T18:59:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 1 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 2 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: 3' 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836554 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836554 2024-09-17T18:59:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554' 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers./ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893122 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893122 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-408021893122' 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:59:18.060 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416397 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416397 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-408021893122 2-339302416397' 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893122 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893122 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836553 1-408021893122 2-339302416397 3-408021893122' 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836553 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836553 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836553 2024-09-17T18:59:18.061 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: waiting osd.0 seq 21474836553 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836553 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836553 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-408021893122 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-408021893122 2024-09-17T18:59:18.062 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:18.338 INFO:tasks.workunit.client.0.smithi105.stderr:175744sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:18.338 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T18:59:18.338 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893123 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893123 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-408021893123' 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416398 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416398 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-408021893123 2-339302416398' 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893123 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893123 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836554 1-408021893123 2-339302416398 3-408021893123' 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:18.339 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836554 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836554 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836554 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: waiting osd.0 seq 21474836554 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836552 -lt 21474836554 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836554 2024-09-17T18:59:18.340 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:18.341 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-408021893123 2024-09-17T18:59:18.341 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:18.341 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T18:59:18.341 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-408021893123 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubu: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893122 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 408021893122' 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: waiting osd.1 seq 408021893122 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893123 -lt 408021893122 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-339302416397 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:59:19.569 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-339302416397 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416397 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 339302416397' 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: waiting osd.2 seq 339302416397 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416398 -lt 339302416397 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-408021893122 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-408021893122 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893122 2024-09-17T18:59:19.570 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 408021893122' 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: waiting osd.3 seq 408021893122 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893123 -lt 408021893122 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:19.571 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(cntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893123 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 408021893123' 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: waiting osd.1 seq 408021893123 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893123 -lt 408021893123 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-339302416398 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-339302416398 2024-09-17T18:59:19.820 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416398 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 339302416398' 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: waiting osd.2 seq 339302416398 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416398 -lt 339302416398 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-408021893123 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-408021893123 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893123 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 408021893123' 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: waiting osd.3 seq 408021893123 2024-09-17T18:59:19.821 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893123 -lt 408021893123 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:19.822 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:21.088 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cepontains("stale") | not)' 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T18:59:21.089 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:59:21.090 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:59:21.091 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:59:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:htest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:59:21.385 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:21.386 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 5 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T18:59:21.387 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T18:59:21.388 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T18:59:21.388 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T18:59:21.388 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ce175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:59:22.343 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:59:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:59:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:175744: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:59:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:59:22.593 INFO:tasks.workunit.client.0.smithi105.stderr:ph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T18:59:22.593 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T18:59:22.593 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=2 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 2 '!=' null 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T18:59:22.594 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:175741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:175741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T18:59:22.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 175740 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T18:59:22.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T18:59:22.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:59:22.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:59:22.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:59:22.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:59:22.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:59:22.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:59:22.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:59:22.975 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:59:22.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:59:22.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:59:22.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:59:22.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:59:22.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:59:22.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:59:22.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:59:22.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:59:22.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:59:22.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:59:23.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:59:23.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:23.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:23.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:59:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:59:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:59:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T18:59:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T18:59:23.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T18:59:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T18:59:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T18:59:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T18:59:23.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T18:59:23.022 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T18:59:23.022 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T18:59:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T18:59:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T18:59:23.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T18:59:23.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T18:59:23.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T18:59:23.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T18:59:23.028 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T18:59:23.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T18:59:23.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T18:59:23.029 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T18:59:23.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:59:23.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T18:59:23.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T18:59:23.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T18:59:23.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T18:59:23.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T18:59:23.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T18:59:23.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T18:59:23.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:23.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:23.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T18:59:23.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T18:59:23.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T18:59:23.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T18:59:23.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T18:59:23.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:23.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:23.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T18:59:23.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-17T18:59:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T18:59:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T18:59:23.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T18:59:23.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:59:23.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:59:23.504 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:59:23.504 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:23.505 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:23.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:23.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T18:59:23.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T18:59:23.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T18:59:23.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:59:23.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:59:23.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T18:59:23.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T18:59:23.715 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:59:23.716 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:59:23.716 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:59:23.717 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:59:23.717 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:23.717 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:23.717 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:59:23.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:59:23.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T18:59:23.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T18:59:23.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T18:59:23.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T18:59:23.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T18:59:23.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T18:59:23.829 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T18:59:23.829 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T18:59:23.829 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T18:59:23.829 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T18:59:23.829 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:23.830 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:23.830 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T18:59:23.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T18:59:23.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T18:59:23.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:24.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:24.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:59:24.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T18:59:24.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-17T18:59:24.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T18:59:24.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T18:59:24.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T18:59:24.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T18:59:24.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f280da07-90ec-45e4-95ce-86c5cec51f01 2024-09-17T18:59:24.839 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 f280da07-90ec-45e4-95ce-86c5cec51f01 2024-09-17T18:59:24.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f280da07-90ec-45e4-95ce-86c5cec51f01' 2024-09-17T18:59:24.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T18:59:24.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCM0elmHKj+MhAAK+I8/d239DZSnYKim2nRZQ== 2024-09-17T18:59:24.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCM0elmHKj+MhAAK+I8/d239DZSnYKim2nRZQ=="}' 2024-09-17T18:59:24.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f280da07-90ec-45e4-95ce-86c5cec51f01 -i td/osd-scrub-repair/0/new.json 2024-09-17T18:59:25.092 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:59:25.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T18:59:25.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCM0elmHKj+MhAAK+I8/d239DZSnYKim2nRZQ== --osd-uuid f280da07-90ec-45e4-95ce-86c5cec51f01 2024-09-17T18:59:25.149 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:25.145+0000 7f15f5f8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:25.151 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:25.149+0000 7f15f5f8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:25.152 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:25.149+0000 7f15f5f8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:25.153 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:25.149+0000 7f15f5f8e7c0 -1 bdev(0x55d7fc58dc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T18:59:25.153 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:25.149+0000 7f15f5f8e7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T18:59:31.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T18:59:31.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T18:59:31.489 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T18:59:31.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T18:59:31.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T18:59:31.899 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T18:59:31.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T18:59:31.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T18:59:31.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T18:59:31.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T18:59:31.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T18:59:31.933 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:31.929+0000 7fc6e4d0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:31.958 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:31.957+0000 7fc6e4d0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:31.959 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:31.957+0000 7fc6e4d0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T18:59:32.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T18:59:32.223 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T18:59:32.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:32.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:32.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:32.988 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:32.985+0000 7fc6e4d0b7c0 -1 Falling back to public interface 2024-09-17T18:59:33.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:33.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:33.560 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T18:59:33.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T18:59:33.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:33.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:33.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:34.893 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:34.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:34.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T18:59:34.894 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T18:59:34.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:34.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:35.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:35.830 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T18:59:35.825+0000 7fc6e4d0b7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T18:59:36.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:36.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:36.220 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T18:59:36.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T18:59:36.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:36.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:37.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:37.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:37.557 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T18:59:37.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T18:59:37.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:37.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:37.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:38.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:38.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:38.891 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T18:59:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T18:59:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:39.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:40.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:40.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:40.228 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T18:59:40.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T18:59:40.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:40.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:40.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:41.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:41.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:41.700 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T18:59:41.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T18:59:41.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:41.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:42.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:43.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:43.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:43.037 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T18:59:43.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T18:59:43.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:43.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:43.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:44.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:44.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:44.363 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T18:59:44.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T18:59:44.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:44.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:44.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:45.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:45.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:45.735 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T18:59:45.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T18:59:45.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:45.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:46.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:47.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:47.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:47.076 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T18:59:47.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T18:59:47.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:47.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:47.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:48.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:48.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:48.461 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T18:59:48.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T18:59:48.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:48.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:49.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:50.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:50.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:50.162 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T18:59:50.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T18:59:50.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:50.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:50.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:51.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:51.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:51.887 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T18:59:51.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T18:59:51.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:51.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:52.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:53.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:53.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:53.674 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T18:59:53.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T18:59:53.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:53.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:54.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:55.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:55.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:55.407 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T18:59:55.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T18:59:55.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:55.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:55.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:56.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:56.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:56.736 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T18:59:56.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T18:59:56.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:56.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:57.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:58.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:58.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:58.531 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T18:59:58.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T18:59:58.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:58.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T18:59:58.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T18:59:59.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T18:59:59.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T18:59:59.875 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T18:59:59.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T18:59:59.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T18:59:59.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:00:00.245 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4171595465,v1:127.0.0.1:6803/4171595465] [v2:127.0.0.1:6804/4171595465,v1:127.0.0.1:6805/4171595465] exists,up f280da07-90ec-45e4-95ce-86c5cec51f01 2024-09-17T19:00:00.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:00:00.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:00:00.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:00:00.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:00:00.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:00:00.251 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:00:00.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:00:00.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:00:00.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:00:00.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6ef7d851-f365-4eb4-8b57-bec530143964 2024-09-17T19:00:00.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6ef7d851-f365-4eb4-8b57-bec530143964' 2024-09-17T19:00:00.255 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 6ef7d851-f365-4eb4-8b57-bec530143964 2024-09-17T19:00:00.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:00:00.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCw0elmm3lFEBAA7Gwmy9qbd0eCKhOrpuO94A== 2024-09-17T19:00:00.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCw0elmm3lFEBAA7Gwmy9qbd0eCKhOrpuO94A=="}' 2024-09-17T19:00:00.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6ef7d851-f365-4eb4-8b57-bec530143964 -i td/osd-scrub-repair/1/new.json 2024-09-17T19:00:00.910 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:00:00.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T19:00:00.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCw0elmm3lFEBAA7Gwmy9qbd0eCKhOrpuO94A== --osd-uuid 6ef7d851-f365-4eb4-8b57-bec530143964 2024-09-17T19:00:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:00.964+0000 7f89e90e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:01.012 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:01.008+0000 7f89e90e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:01.014 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:01.012+0000 7f89e90e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:01.015 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:01.012+0000 7f89e90e27c0 -1 bdev(0x56222240fc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T19:00:01.015 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:01.012+0000 7f89e90e27c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T19:00:07.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T19:00:07.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:00:07.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T19:00:07.490 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T19:00:07.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:00:07.941 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T19:00:07.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T19:00:07.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:00:07.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:00:07.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:00:07.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:00:07.973 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:07.972+0000 7f514b5c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:07.983 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:07.980+0000 7f514b5c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:07.984 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:07.980+0000 7f514b5c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:08.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T19:00:08.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:00:08.276 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:08.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:08.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:09.292 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:09.288+0000 7f514b5c07c0 -1 Falling back to public interface 2024-09-17T19:00:09.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:09.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:09.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:00:09.614 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:00:09.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:09.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:09.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:10.934 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:00:10.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:00:10.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:10.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:11.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:12.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:12.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:12.273 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:00:12.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:00:12.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:12.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:12.343 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:12.340+0000 7f514b5c07c0 -1 osd.1 0 log_to_monitors true 2024-09-17T19:00:12.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:13.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:13.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:13.592 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:00:13.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:00:13.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:13.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:13.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:14.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:14.970 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:00:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:00:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:14.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:15.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:16.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:16.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:16.393 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:00:16.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:00:16.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:16.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:16.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:17.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:17.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:17.711 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:00:17.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:00:17.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:17.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:18.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:19.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:19.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:19.051 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:00:19.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:00:19.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:19.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:19.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:20.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:20.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:20.455 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:00:20.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:00:20.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:20.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:20.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:21.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:21.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:21.792 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:00:21.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:00:21.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:21.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:22.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:23.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:23.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:23.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:00:23.186 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:00:23.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:23.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:23.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:24.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:24.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:24.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:00:24.522 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:00:24.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:24.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:24.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:25.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:25.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:25.849 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:00:25.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:00:25.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:25.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:26.400 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:27.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:27.402 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:00:27.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:27.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:00:27.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:27.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:27.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:28.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:28.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:28.793 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:00:28.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:00:28.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:28.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:29.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:30.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:30.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:30.442 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:00:30.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:00:30.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:30.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:31.240 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:32.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:32.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:32.242 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:00:32.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:00:32.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:32.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:32.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:00:33.978 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:00:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:33.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:34.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:35.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:35.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:35.913 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:00:35.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:00:35.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:35.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:36.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:37.229 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:00:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:00:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:00:37.557 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3735785155,v1:127.0.0.1:6811/3735785155] [v2:127.0.0.1:6812/3735785155,v1:127.0.0.1:6813/3735785155] exists,up 6ef7d851-f365-4eb4-8b57-bec530143964 2024-09-17T19:00:37.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:00:37.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:00:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:00:37.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:00:37.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:00:37.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:00:37.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:00:37.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=318ddcba-2e8c-4bad-a3ce-5c8e2011f881 2024-09-17T19:00:37.564 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 318ddcba-2e8c-4bad-a3ce-5c8e2011f881 2024-09-17T19:00:37.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 318ddcba-2e8c-4bad-a3ce-5c8e2011f881' 2024-09-17T19:00:37.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:00:37.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDV0elmqQPZIhAAnAF+iQGHf/LzzXeQiXPgpA== 2024-09-17T19:00:37.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDV0elmqQPZIhAAnAF+iQGHf/LzzXeQiXPgpA=="}' 2024-09-17T19:00:37.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 318ddcba-2e8c-4bad-a3ce-5c8e2011f881 -i td/osd-scrub-repair/2/new.json 2024-09-17T19:00:38.048 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:00:38.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T19:00:38.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDV0elmqQPZIhAAnAF+iQGHf/LzzXeQiXPgpA== --osd-uuid 318ddcba-2e8c-4bad-a3ce-5c8e2011f881 2024-09-17T19:00:38.104 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:38.100+0000 7f92cc1d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:38.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:38.140+0000 7f92cc1d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:38.143 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:38.140+0000 7f92cc1d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:38.144 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:38.140+0000 7f92cc1d27c0 -1 bdev(0x558b23155c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T19:00:38.144 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:38.140+0000 7f92cc1d27c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T19:00:44.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T19:00:44.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:00:44.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T19:00:44.360 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T19:00:44.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:00:44.758 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T19:00:44.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T19:00:44.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:00:44.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:00:44.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:00:44.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:00:44.785 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:44.779+0000 7fb13fc497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:44.787 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:44.783+0000 7fb13fc497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:44.788 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:44.783+0000 7fb13fc497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:00:45.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T19:00:45.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:00:45.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:00:45.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:00:45.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:00:45.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:45.117 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:00:45.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:00:45.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:45.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:45.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:46.080 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:46.075+0000 7fb13fc497c0 -1 Falling back to public interface 2024-09-17T19:00:46.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:46.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:46.464 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:00:46.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:00:46.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:46.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:46.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:47.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:47.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:47.795 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:00:47.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:00:47.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:47.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:48.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:49.003 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:00:48.999+0000 7fb13fc497c0 -1 osd.2 0 log_to_monitors true 2024-09-17T19:00:49.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:49.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:49.151 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:00:49.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:00:49.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:49.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:49.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:50.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:50.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:50.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:00:50.523 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:00:50.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:50.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:50.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:51.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:51.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:51.992 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:00:51.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:00:51.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:51.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:52.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:53.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:53.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:53.424 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:00:53.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:00:53.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:53.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:53.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:54.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:54.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:54.895 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:00:54.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:00:54.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:54.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:55.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:56.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:56.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:56.233 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:00:56.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:00:56.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:56.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:56.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:57.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:57.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:57.618 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:00:57.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:00:57.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:57.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:58.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:00:59.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:00:59.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:00:59.057 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:00:59.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:00:59.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:00:59.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:00:59.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:00.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:00.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:00.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:01:00.393 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:01:00.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:00.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:00.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:01.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:01.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:01.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:01:01.787 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:01:01.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:01.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:02.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:03.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:03.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:03.386 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:01:03.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:01:03.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:03.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:04.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:05.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:05.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:05.149 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:01:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:01:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:05.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:05.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:06.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:06.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:06.635 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:01:06.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:01:06.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:06.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:07.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:08.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:08.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:08.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:01:08.245 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:01:08.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:08.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:08.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:09.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:09.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:09.992 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:01:09.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:01:09.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:09.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:10.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:11.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:11.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:11.813 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:01:11.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:01:11.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:11.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:12.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:13.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:13.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:13.699 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:01:13.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:01:13.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:13.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:01:14.032 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2039813489,v1:127.0.0.1:6819/2039813489] [v2:127.0.0.1:6820/2039813489,v1:127.0.0.1:6821/2039813489] exists,up 318ddcba-2e8c-4bad-a3ce-5c8e2011f881 2024-09-17T19:01:14.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:01:14.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:01:14.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:01:14.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:01:14.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:01:14.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:01:14.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:01:14.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:01:14.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:01:14.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:01:14.040 INFO:tasks.workunit.client.0.smithi105.stdout:add osd3 4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:01:14.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4846fc06-1303-474b-b310-d67da1bf6660' 2024-09-17T19:01:14.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:01:14.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD60elm3OCmAxAAWDBd+skySPis//gewTThLw== 2024-09-17T19:01:14.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD60elm3OCmAxAAWDBd+skySPis//gewTThLw=="}' 2024-09-17T19:01:14.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4846fc06-1303-474b-b310-d67da1bf6660 -i td/osd-scrub-repair/3/new.json 2024-09-17T19:01:14.440 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:01:14.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-17T19:01:14.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD60elm3OCmAxAAWDBd+skySPis//gewTThLw== --osd-uuid 4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:01:14.492 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:14.487+0000 7fadbf5397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:14.542 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:14.539+0000 7fadbf5397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:14.543 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:14.539+0000 7fadbf5397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:14.544 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:14.539+0000 7fadbf5397c0 -1 bdev(0x55e07e759c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-17T19:01:14.544 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:14.539+0000 7fadbf5397c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-17T19:01:21.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-17T19:01:21.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:01:21.422 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd3 key to auth repository 2024-09-17T19:01:21.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T19:01:21.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:01:21.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T19:01:21.830 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.3 2024-09-17T19:01:21.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:01:21.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:01:21.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:01:21.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:01:21.862 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:21.859+0000 7f86b6e127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:21.864 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:21.859+0000 7f86b6e127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:21.865 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:21.859+0000 7f86b6e127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:22.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T19:01:22.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:01:22.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:01:22.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:01:22.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:01:22.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:22.171 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:01:22.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:01:22.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:22.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:22.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:23.156 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:23.151+0000 7f86b6e127c0 -1 Falling back to public interface 2024-09-17T19:01:23.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:23.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:23.504 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:01:23.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:01:23.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:23.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:23.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:24.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:24.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:24.829 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:01:24.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:01:24.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:24.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:25.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:26.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:26.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:26.162 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:01:26.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:01:26.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:26.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:26.191+0000 7f86b6e127c0 -1 osd.3 0 log_to_monitors true 2024-09-17T19:01:26.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:27.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:27.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:27.563 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:01:27.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:01:27.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:27.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:27.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:28.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:28.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:28.921 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:01:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:01:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:29.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:30.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:30.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:30.292 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:01:30.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:01:30.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:30.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:30.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:31.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:31.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:31.625 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:01:31.626 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:01:31.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:31.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:32.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:33.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:33.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:33.130 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:01:33.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:01:33.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:33.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:33.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:34.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:34.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:34.457 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:01:34.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:01:34.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:34.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:34.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:35.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:35.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:35.776 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:01:35.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:01:35.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:35.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:36.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:37.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:37.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:37.149 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:01:37.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:01:37.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:37.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:37.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:38.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:38.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:38.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:01:38.485 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:01:38.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:38.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:38.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:39.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:39.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:39.813 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:01:39.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:01:39.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:39.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:40.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:41.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:41.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:41.510 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:01:41.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:01:41.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:41.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:42.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:43.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:43.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:43.244 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:01:43.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:01:43.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:43.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:43.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:44.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:44.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:44.878 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:01:44.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:01:44.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:44.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:45.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:46.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:46.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:46.621 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:01:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:01:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:47.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:48.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:48.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:48.246 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:01:48.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:01:48.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:48.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:48.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:49.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:49.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:49.838 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:01:49.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:01:49.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:49.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:50.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:51.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:01:51.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:51.200 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:01:51.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:01:51.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:51.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:01:51.538 INFO:tasks.workunit.client.0.smithi105.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1952252327,v1:127.0.0.1:6827/1952252327] [v2:127.0.0.1:6828/1952252327,v1:127.0.0.1:6829/1952252327] exists,up 4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-17T19:01:51.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:01:51.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:01:51.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:01:51.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=940e2086-ceed-4138-87d0-2aadb4ace236 2024-09-17T19:01:51.545 INFO:tasks.workunit.client.0.smithi105.stdout:add osd4 940e2086-ceed-4138-87d0-2aadb4ace236 2024-09-17T19:01:51.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 940e2086-ceed-4138-87d0-2aadb4ace236' 2024-09-17T19:01:51.545 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:01:51.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAf0ulmDiiLIRAAph/Mplm5P5fShwl/GD7Org== 2024-09-17T19:01:51.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAf0ulmDiiLIRAAph/Mplm5P5fShwl/GD7Org=="}' 2024-09-17T19:01:51.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 940e2086-ceed-4138-87d0-2aadb4ace236 -i td/osd-scrub-repair/4/new.json 2024-09-17T19:01:51.930 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:01:51.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-17T19:01:51.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAf0ulmDiiLIRAAph/Mplm5P5fShwl/GD7Org== --osd-uuid 940e2086-ceed-4138-87d0-2aadb4ace236 2024-09-17T19:01:51.983 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:51.978+0000 7f8f31ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:52.023 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:52.018+0000 7f8f31ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:52.025 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:52.022+0000 7f8f31ed87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:52.026 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:52.022+0000 7f8f31ed87c0 -1 bdev(0x55ec8d2a9c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-17T19:01:52.026 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:52.022+0000 7f8f31ed87c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-17T19:01:58.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-17T19:01:58.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:01:58.154 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd4 key to auth repository 2024-09-17T19:01:58.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T19:01:58.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:01:58.568 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.4 2024-09-17T19:01:58.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T19:01:58.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:01:58.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:01:58.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:01:58.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:01:58.601 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:58.594+0000 7f192902d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:58.630 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:58.626+0000 7f192902d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:58.631 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:58.626+0000 7f192902d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:01:58.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T19:01:58.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:01:58.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T19:01:58.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:01:58.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:01:58.928 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:01:58.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:01:58.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:01:58.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:01:58.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:01:59.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:01:59.924 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:01:59.918+0000 7f192902d7c0 -1 Falling back to public interface 2024-09-17T19:02:00.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:00.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:00.274 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:02:00.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:02:00.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:00.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:01.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:01.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:01.591 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:02:01.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:02:01.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:01.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:01.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:02.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:02.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:02.912 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:02:02.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:02:02.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:02.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:02.938 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:02.934+0000 7f192902d7c0 -1 osd.4 0 log_to_monitors true 2024-09-17T19:02:03.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:04.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:04.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:04.301 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:02:04.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:02:04.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:04.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:04.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:05.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:05.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:05.682 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:02:05.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:02:05.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:05.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:06.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:07.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:07.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:07.145 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:02:07.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:02:07.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:07.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:07.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:08.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:08.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:08.467 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:02:08.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:02:08.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:08.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:08.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:09.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:09.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:09.845 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:02:09.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:02:09.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:09.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:10.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:11.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:11.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:11.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:02:11.335 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:02:11.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:11.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:11.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:12.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:12.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:12.878 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:02:12.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:02:12.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:12.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:13.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:14.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:14.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:14.213 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:02:14.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:02:14.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:14.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:14.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:15.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:15.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:15.614 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:02:15.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:02:15.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:15.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:16.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:17.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:17.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:17.209 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:02:17.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:02:17.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:17.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:18.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:19.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:19.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:19.009 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:02:19.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:02:19.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:19.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:19.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:20.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:20.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:20.680 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:02:20.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:02:20.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:20.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:21.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:22.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:22.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:22.209 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:02:22.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:02:22.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:22.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:22.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:23.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:23.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:23.925 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:02:23.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:02:23.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:23.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:24.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:25.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:25.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:25.727 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:02:25.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:02:25.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:25.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:26.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:27.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:27.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:27.599 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:02:27.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:02:27.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:27.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:02:28.089 INFO:tasks.workunit.client.0.smithi105.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/228144024,v1:127.0.0.1:6835/228144024] [v2:127.0.0.1:6836/228144024,v1:127.0.0.1:6837/228144024] exists,up 940e2086-ceed-4138-87d0-2aadb4ace236 2024-09-17T19:02:28.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:02:28.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:02:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:02:28.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:02:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 ef4552e6-877d-496f-90d6-eb71ad7992f3' 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stdout:add osd5 ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:02:28.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:02:28.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBE0ulmJY/LBhAAsWLBOK+zj+vz0RlRrxyiEA== 2024-09-17T19:02:28.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBE0ulmJY/LBhAAsWLBOK+zj+vz0RlRrxyiEA=="}' 2024-09-17T19:02:28.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ef4552e6-877d-496f-90d6-eb71ad7992f3 -i td/osd-scrub-repair/5/new.json 2024-09-17T19:02:28.510 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:02:28.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-17T19:02:28.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBE0ulmJY/LBhAAsWLBOK+zj+vz0RlRrxyiEA== --osd-uuid ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:02:28.562 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:28.562+0000 7f38f51e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:02:28.595 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:28.594+0000 7f38f51e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:02:28.596 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:28.594+0000 7f38f51e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:02:28.597 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:28.594+0000 7f38f51e57c0 -1 bdev(0x558d6d46bc00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-17T19:02:28.597 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:28.594+0000 7f38f51e57c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-17T19:02:35.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-17T19:02:35.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:02:35.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T19:02:35.317 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd5 key to auth repository 2024-09-17T19:02:35.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:02:35.711 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.5 2024-09-17T19:02:35.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T19:02:35.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:02:35.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:02:35.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:02:35.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:02:35.744 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:35.742+0000 7f40f226b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:02:35.746 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:35.746+0000 7f40f226b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:02:35.747 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:35.746+0000 7f40f226b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:02:36.075 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:02:36.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:36.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:36.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:37.264 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:37.261+0000 7f40f226b7c0 -1 Falling back to public interface 2024-09-17T19:02:37.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:37.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:37.404 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:02:37.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:02:37.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:37.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:37.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:38.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:38.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:38.739 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:02:38.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:02:38.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:38.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:39.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:40.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:40.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:40.069 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:02:40.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:02:40.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:40.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:40.179 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:02:40.177+0000 7f40f226b7c0 -1 osd.5 0 log_to_monitors true 2024-09-17T19:02:40.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:41.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:41.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:41.424 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:02:41.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:02:41.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:41.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:41.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:42.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:42.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:42.762 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:02:42.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:02:42.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:42.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:43.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:44.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:44.225 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:02:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:02:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:44.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:45.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:45.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:45.723 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:02:45.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:02:45.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:45.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:46.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:47.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:47.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:47.075 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:02:47.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:02:47.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:47.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:47.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:48.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:48.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:48.411 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:02:48.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:02:48.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:48.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:48.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:49.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:49.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:49.753 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:02:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:02:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:49.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:50.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:51.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:51.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:51.182 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:02:51.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:02:51.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:51.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:51.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:52.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:52.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:52.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:02:52.519 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:02:52.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:52.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:52.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:53.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:53.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:53.856 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:02:53.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:02:53.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:53.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:54.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:55.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:55.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:55.404 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:02:55.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:02:55.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:55.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:55.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:56.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:56.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:56.854 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:02:56.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:02:56.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:56.855 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:57.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:02:58.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:02:58.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:02:58.518 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:02:58.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:02:58.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:02:58.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:02:59.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:00.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:00.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:00.379 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:03:00.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:03:00.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:00.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:03:00.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:01.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:01.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:01.993 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:03:01.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:03:01.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:01.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:03:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:03.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:03.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:03.370 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:03:03.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:03:03.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:03.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:03:03.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:04.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:04.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:04.916 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:03:04.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:03:04.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:04.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:03:05.248 INFO:tasks.workunit.client.0.smithi105.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1382874822,v1:127.0.0.1:6843/1382874822] [v2:127.0.0.1:6844/1382874822,v1:127.0.0.1:6845/1382874822] exists,up ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:03:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:03:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:03:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:03:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:03:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-17T19:03:05.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-17T19:03:05.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:03:05.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-17T19:03:05.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:03:05.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a603cbea-cc76-49ee-9e0a-e3d2c51c41fc 2024-09-17T19:03:05.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 a603cbea-cc76-49ee-9e0a-e3d2c51c41fc' 2024-09-17T19:03:05.256 INFO:tasks.workunit.client.0.smithi105.stdout:add osd6 a603cbea-cc76-49ee-9e0a-e3d2c51c41fc 2024-09-17T19:03:05.256 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:03:05.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBp0ulmv+6FEBAAeoy3Ojv3BRragXyvamEt/g== 2024-09-17T19:03:05.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBp0ulmv+6FEBAAeoy3Ojv3BRragXyvamEt/g=="}' 2024-09-17T19:03:05.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a603cbea-cc76-49ee-9e0a-e3d2c51c41fc -i td/osd-scrub-repair/6/new.json 2024-09-17T19:03:05.668 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:03:05.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-17T19:03:05.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBp0ulmv+6FEBAAeoy3Ojv3BRragXyvamEt/g== --osd-uuid a603cbea-cc76-49ee-9e0a-e3d2c51c41fc 2024-09-17T19:03:05.725 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:05.725+0000 7f1c0cd437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:05.762 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:05.761+0000 7f1c0cd437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:05.763 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:05.761+0000 7f1c0cd437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:05.764 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:05.761+0000 7f1c0cd437c0 -1 bdev(0x565327f83c00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-17T19:03:05.764 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:05.761+0000 7f1c0cd437c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-17T19:03:11.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-17T19:03:11.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:03:11.573 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd6 key to auth repository 2024-09-17T19:03:11.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T19:03:11.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:03:11.984 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.6 2024-09-17T19:03:11.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T19:03:11.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:03:11.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:03:11.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:03:11.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:03:12.017 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:12.013+0000 7f866e8647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:12.019 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:12.017+0000 7f866e8647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:12.020 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:12.017+0000 7f866e8647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:12.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:03:12.302 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:03:12.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:12.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:12.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:13.344 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:13.341+0000 7f866e8647c0 -1 Falling back to public interface 2024-09-17T19:03:13.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:13.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:13.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:03:13.636 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:03:13.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:13.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:13.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:14.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:14.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:14.973 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:03:14.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:03:14.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:14.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:15.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:16.211 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:16.209+0000 7f866e8647c0 -1 osd.6 0 log_to_monitors true 2024-09-17T19:03:16.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:16.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:16.310 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:03:16.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:03:16.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:16.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:16.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:17.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:17.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:17.648 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:03:17.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:03:17.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:17.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:17.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:18.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:18.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:18.986 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:03:18.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:03:18.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:18.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:19.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:20.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:20.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:20.344 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:03:20.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:03:20.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:20.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:20.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:21.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:21.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:21.680 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:03:21.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:03:21.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:21.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:22.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:23.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:23.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:23.016 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:03:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:03:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:23.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:23.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:24.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:24.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:24.348 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:03:24.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:03:24.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:24.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:24.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:25.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:25.686 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:03:25.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:25.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:03:25.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:25.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:26.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:27.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:27.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:27.032 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:03:27.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:03:27.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:27.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:27.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:28.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:28.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:28.447 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:03:28.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:03:28.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:28.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:28.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:29.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:29.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:29.784 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:03:29.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:03:29.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:29.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:30.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:31.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:31.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:31.406 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:03:31.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:03:31.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:31.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:32.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:33.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:33.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:33.457 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:03:33.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:03:33.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:33.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:34.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:35.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:35.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:35.132 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:03:35.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:03:35.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:35.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:35.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:36.863 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:03:36.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:03:36.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:36.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:37.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:38.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:38.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:38.217 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:03:38.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:03:38.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:38.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:39.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:40.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:40.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:40.125 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:03:40.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:03:40.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:40.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:41.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:42.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:42.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:42.068 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:03:42.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:03:42.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:42.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:03:42.545 INFO:tasks.workunit.client.0.smithi105.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2034779252,v1:127.0.0.1:6851/2034779252] [v2:127.0.0.1:6852/2034779252,v1:127.0.0.1:6853/2034779252] exists,up a603cbea-cc76-49ee-9e0a-e3d2c51c41fc 2024-09-17T19:03:42.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:03:42.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:03:42.551 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:03:42.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:03:42.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:03:42.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:03:42.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:03:42.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-17T19:03:42.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:03:42.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1867c97b-c75e-4a60-a829-67a1beb5de22 2024-09-17T19:03:42.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 1867c97b-c75e-4a60-a829-67a1beb5de22' 2024-09-17T19:03:42.554 INFO:tasks.workunit.client.0.smithi105.stdout:add osd7 1867c97b-c75e-4a60-a829-67a1beb5de22 2024-09-17T19:03:42.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:03:42.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCO0ulmmv5VIhAApvDyDsimICOypJtV4SSmHg== 2024-09-17T19:03:42.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCO0ulmmv5VIhAApvDyDsimICOypJtV4SSmHg=="}' 2024-09-17T19:03:42.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1867c97b-c75e-4a60-a829-67a1beb5de22 -i td/osd-scrub-repair/7/new.json 2024-09-17T19:03:42.969 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:03:42.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-17T19:03:42.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCO0ulmmv5VIhAApvDyDsimICOypJtV4SSmHg== --osd-uuid 1867c97b-c75e-4a60-a829-67a1beb5de22 2024-09-17T19:03:43.027 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:43.024+0000 7f8ddaf147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:43.029 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:43.028+0000 7f8ddaf147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:43.030 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:43.028+0000 7f8ddaf147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:43.031 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:43.028+0000 7f8ddaf147c0 -1 bdev(0x560ba7ea5c00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-17T19:03:43.031 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:43.028+0000 7f8ddaf147c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-17T19:03:49.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-17T19:03:49.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:03:49.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-17T19:03:49.718 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd7 key to auth repository 2024-09-17T19:03:49.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:03:50.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-17T19:03:50.125 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.7 2024-09-17T19:03:50.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:03:50.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:03:50.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:03:50.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:03:50.156 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:50.152+0000 7f1501b957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:50.158 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:50.156+0000 7f1501b957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:50.159 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:50.156+0000 7f1501b957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:03:50.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-17T19:03:50.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:03:50.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-17T19:03:50.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:03:50.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:03:50.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:50.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:03:50.448 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:03:50.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:50.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:50.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:51.408 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:51.404+0000 7f1501b957c0 -1 Falling back to public interface 2024-09-17T19:03:51.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:51.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:51.776 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:03:51.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:03:51.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:51.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:52.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:53.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:53.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:53.112 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:03:53.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:03:53.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:53.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:53.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:54.313 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:03:54.308+0000 7f1501b957c0 -1 osd.7 0 log_to_monitors true 2024-09-17T19:03:54.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:54.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:54.443 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:03:54.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:03:54.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:54.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:54.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:55.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:55.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:55.840 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:03:55.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:03:55.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:55.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:56.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:57.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:57.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:57.170 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:03:57.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:03:57.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:57.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:57.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:58.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:58.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:58.506 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:03:58.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:03:58.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:58.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:03:58.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:03:59.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:03:59.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:03:59.842 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:03:59.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:03:59.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:03:59.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:00.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:01.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:01.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:01.213 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:04:01.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:04:01.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:01.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:01.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:02.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:02.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:02.546 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:04:02.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:04:02.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:02.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:02.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:03.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:03.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:03.938 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:04:03.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:04:03.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:03.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:04.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:05.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:05.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:05.375 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:04:05.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:04:05.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:05.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:05.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:06.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:06.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:06.733 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:04:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:04:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:06.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:07.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:08.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:08.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:08.166 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:04:08.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:04:08.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:08.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:09.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:10.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:10.117 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:04:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:04:10.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:10.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:11.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:11.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:11.463 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:04:11.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:04:11.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:11.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:11.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:12.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:12.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:12.912 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:04:12.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:04:12.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:12.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:13.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:14.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:14.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:14.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:04:14.511 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:04:14.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:14.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:15.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:16.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:16.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:16.360 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:04:16.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:04:16.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:16.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:16.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:17.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:17.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:17.714 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:04:17.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:04:17.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:17.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:18.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:19.204 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:04:19.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:04:19.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:19.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:04:19.560 INFO:tasks.workunit.client.0.smithi105.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/621703562,v1:127.0.0.1:6859/621703562] [v2:127.0.0.1:6860/621703562,v1:127.0.0.1:6861/621703562] exists,up 1867c97b-c75e-4a60-a829-67a1beb5de22 2024-09-17T19:04:19.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:04:19.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:04:19.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:04:19.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-17T19:04:19.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:04:19.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:04:19.565 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-17T19:04:19.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:04:19.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6bdeecf1-1588-4608-bc20-112662ce473b 2024-09-17T19:04:19.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 6bdeecf1-1588-4608-bc20-112662ce473b' 2024-09-17T19:04:19.566 INFO:tasks.workunit.client.0.smithi105.stdout:add osd8 6bdeecf1-1588-4608-bc20-112662ce473b 2024-09-17T19:04:19.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:04:19.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCz0ulmGaj/IhAARcmYtwVkuvRnbNNYVhaMWQ== 2024-09-17T19:04:19.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCz0ulmGaj/IhAARcmYtwVkuvRnbNNYVhaMWQ=="}' 2024-09-17T19:04:19.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6bdeecf1-1588-4608-bc20-112662ce473b -i td/osd-scrub-repair/8/new.json 2024-09-17T19:04:20.039 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:04:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-17T19:04:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCz0ulmGaj/IhAARcmYtwVkuvRnbNNYVhaMWQ== --osd-uuid 6bdeecf1-1588-4608-bc20-112662ce473b 2024-09-17T19:04:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:20.092+0000 7ff0751a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:20.139 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:20.136+0000 7ff0751a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:20.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:20.136+0000 7ff0751a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:20.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:20.140+0000 7ff0751a97c0 -1 bdev(0x55ad05f3fc00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-17T19:04:20.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:20.140+0000 7ff0751a97c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-17T19:04:27.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-17T19:04:27.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:04:27.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-17T19:04:27.204 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd8 key to auth repository 2024-09-17T19:04:27.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:04:27.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-17T19:04:27.598 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.8 2024-09-17T19:04:27.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:04:27.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:04:27.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:04:27.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:04:27.629 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:27.627+0000 7f1cc95527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:27.632 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:27.627+0000 7f1cc95527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:27.633 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:27.627+0000 7f1cc95527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:27.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-17T19:04:27.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:04:27.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-17T19:04:27.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:04:27.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:04:27.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:27.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:04:27.977 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:04:27.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:27.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:28.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:28.924 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:28.919+0000 7f1cc95527c0 -1 Falling back to public interface 2024-09-17T19:04:29.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:29.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:29.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:04:29.305 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:04:29.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:29.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:29.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:30.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:30.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:30.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:04:30.641 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:04:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:30.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:31.917 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:31.911+0000 7f1cc95527c0 -1 osd.8 0 log_to_monitors true 2024-09-17T19:04:31.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:31.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:31.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:04:31.977 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:04:31.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:31.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:32.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:33.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:33.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:33.464 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:04:33.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:04:33.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:33.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:33.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:34.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:34.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:34.822 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:04:34.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:04:34.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:34.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:35.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:36.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:36.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:36.149 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:04:36.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:04:36.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:36.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:36.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:37.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:37.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:37.488 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:04:37.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:04:37.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:37.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:37.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:39.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:39.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:39.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:04:39.001 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:04:39.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:39.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:39.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:40.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:40.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:40.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:04:40.322 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:04:40.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:40.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:40.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:41.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:41.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:41.660 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:04:41.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:04:41.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:41.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:42.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:43.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:43.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:43.024 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:04:43.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:04:43.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:43.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:43.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:44.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:44.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:44.360 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:04:44.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:04:44.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:44.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:44.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:45.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:45.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:45.767 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:04:45.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:04:45.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:45.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:46.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:47.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:47.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:47.772 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:04:47.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:04:47.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:47.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:48.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:49.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:49.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:49.195 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:04:49.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:04:49.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:49.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:50.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:51.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:51.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:51.049 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:04:51.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:04:51.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:51.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:51.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:52.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:52.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:52.868 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:04:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:04:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:53.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:54.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:54.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:54.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:04:54.603 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:04:54.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:54.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:55.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:56.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:56.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:56.002 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:04:56.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:04:56.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:56.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:56.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:04:57.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:04:57.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:04:57.461 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:04:57.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:04:57.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:04:57.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:04:57.783 INFO:tasks.workunit.client.0.smithi105.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/330015717,v1:127.0.0.1:6867/330015717] [v2:127.0.0.1:6868/330015717,v1:127.0.0.1:6869/330015717] exists,up 6bdeecf1-1588-4608-bc20-112662ce473b 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:04:57.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:04:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:04:57.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:04:57.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:04:57.791 INFO:tasks.workunit.client.0.smithi105.stdout:add osd9 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:04:57.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 4bce93e9-bde2-482a-a639-925aec3e7f95' 2024-09-17T19:04:57.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:04:57.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZ0ulmx2JTMBAACoOT6ChGiLc7dEQNqn9OMw== 2024-09-17T19:04:57.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZ0ulmx2JTMBAACoOT6ChGiLc7dEQNqn9OMw=="}' 2024-09-17T19:04:57.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4bce93e9-bde2-482a-a639-925aec3e7f95 -i td/osd-scrub-repair/9/new.json 2024-09-17T19:04:58.243 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:04:58.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-17T19:04:58.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZ0ulmx2JTMBAACoOT6ChGiLc7dEQNqn9OMw== --osd-uuid 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:04:58.298 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:58.295+0000 7f840e0407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:58.364 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:58.359+0000 7f840e0407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:58.366 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:58.363+0000 7f840e0407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:04:58.367 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:58.363+0000 7f840e0407c0 -1 bdev(0x558f9ca23c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-17T19:04:58.367 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:04:58.363+0000 7f840e0407c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-17T19:05:05.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-17T19:05:05.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:05:05.387 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd9 key to auth repository 2024-09-17T19:05:05.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-17T19:05:05.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:05:05.797 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.9 2024-09-17T19:05:05.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-17T19:05:05.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:05:05.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:05:05.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:05:05.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:05:05.828 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:05:05.823+0000 7f659bdd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:05:05.831 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:05:05.827+0000 7f659bdd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:05:05.832 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:05:05.827+0000 7f659bdd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:05:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-17T19:05:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:05:06.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:06.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:06.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:07.124 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:05:07.119+0000 7f659bdd67c0 -1 Falling back to public interface 2024-09-17T19:05:07.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:07.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:07.470 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:05:07.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:05:07.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:07.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:07.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:08.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:08.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:08.790 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:05:08.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:05:08.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:08.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:09.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:10.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:10.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:10.125 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:05:10.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:05:10.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:10.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:10.271 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:05:10.267+0000 7f659bdd67c0 -1 osd.9 0 log_to_monitors true 2024-09-17T19:05:10.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:11.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:11.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:11.495 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:05:11.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:05:11.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:11.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:11.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:12.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:12.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:12.832 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:05:12.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:05:12.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:12.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:13.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:14.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:14.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:14.203 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:05:14.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:05:14.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:14.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:14.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:15.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:15.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:15.675 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:05:15.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:05:15.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:15.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:16.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:17.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:17.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:17.141 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:05:17.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:05:17.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:17.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:17.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:18.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:18.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:18.474 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:05:18.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:05:18.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:18.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:18.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:19.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:19.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:19.809 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:05:19.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:05:19.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:19.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:20.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:21.145 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:05:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:05:21.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:21.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:21.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:22.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:22.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:22.473 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:05:22.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:05:22.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:22.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:22.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:23.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:23.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:23.807 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:05:23.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:05:23.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:23.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:24.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:25.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:25.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:25.347 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:05:25.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:05:25.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:25.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:25.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:26.684 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:05:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:05:26.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:26.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:27.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:28.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:28.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:28.306 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:05:28.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:05:28.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:28.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:28.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:29.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:29.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:29.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:05:29.684 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:05:29.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:29.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:30.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:31.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:31.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:31.167 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:05:31.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:05:31.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:31.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:31.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:32.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:32.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:32.636 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:05:32.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:05:32.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:32.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:33.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:34.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:34.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:34.028 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:05:34.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:05:34.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:34.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:34.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:05:35.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:05:35.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:05:35.819 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T19:05:35.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T19:05:35.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:05:35.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:05:36.157 INFO:tasks.workunit.client.0.smithi105.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1993642249,v1:127.0.0.1:6875/1993642249] [v2:127.0.0.1:6876/1993642249,v1:127.0.0.1:6877/1993642249] exists,up 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:05:36.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:05:36.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:05:36.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:05:36.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-17T19:05:36.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T19:05:36.480 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T19:05:36.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T19:05:36.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T19:05:36.963 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T19:05:36.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:05:37.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T19:05:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-17T19:05:41.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:05:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:05:41.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:05:41.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:05:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:05:41.990 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:05:41.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:05:41.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:05:41.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:05:42.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:05:42.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:05:42.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:05:42.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:05:42.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:05:42.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:05:42.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:05:42.448 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:05:42.448 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:05:42.448 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:05:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:42.450 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:05:42.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836556 2024-09-17T19:05:42.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836556 2024-09-17T19:05:42.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556' 2024-09-17T19:05:42.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:42.600 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:05:42.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673024 2024-09-17T19:05:42.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673024 2024-09-17T19:05:42.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024' 2024-09-17T19:05:42.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:42.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:05:42.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509501 2024-09-17T19:05:42.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509501 2024-09-17T19:05:42.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501' 2024-09-17T19:05:42.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:42.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:05:43.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345973 2024-09-17T19:05:43.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345973 2024-09-17T19:05:43.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973' 2024-09-17T19:05:43.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:43.108 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:05:43.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182446 2024-09-17T19:05:43.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182446 2024-09-17T19:05:43.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973 4-107374182446' 2024-09-17T19:05:43.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:43.251 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:05:43.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018917 2024-09-17T19:05:43.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018917 2024-09-17T19:05:43.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973 4-107374182446 5-128849018917' 2024-09-17T19:05:43.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:43.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:05:43.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855395 2024-09-17T19:05:43.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855395 2024-09-17T19:05:43.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973 4-107374182446 5-128849018917 6-150323855395' 2024-09-17T19:05:43.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:43.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:05:43.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691862 2024-09-17T19:05:43.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691862 2024-09-17T19:05:43.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973 4-107374182446 5-128849018917 6-150323855395 7-171798691862' 2024-09-17T19:05:43.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:43.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:05:43.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528333 2024-09-17T19:05:43.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528333 2024-09-17T19:05:43.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973 4-107374182446 5-128849018917 6-150323855395 7-171798691862 8-193273528333' 2024-09-17T19:05:43.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:43.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:05:43.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364814 2024-09-17T19:05:43.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364814 2024-09-17T19:05:43.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836556 1-42949673024 2-64424509501 3-85899345973 4-107374182446 5-128849018917 6-150323855395 7-171798691862 8-193273528333 9-214748364814' 2024-09-17T19:05:43.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:43.919 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836556 2024-09-17T19:05:43.919 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:43.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:05:43.921 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836556 2024-09-17T19:05:43.921 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:43.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836556 2024-09-17T19:05:43.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-09-17T19:05:43.922 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836556 2024-09-17T19:05:43.923 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:05:44.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836556 2024-09-17T19:05:44.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:44.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673024 2024-09-17T19:05:44.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:44.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:05:44.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673024 2024-09-17T19:05:44.257 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:44.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673024 2024-09-17T19:05:44.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673024' 2024-09-17T19:05:44.258 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949673024 2024-09-17T19:05:44.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:05:44.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673024 -lt 42949673024 2024-09-17T19:05:44.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:44.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509501 2024-09-17T19:05:44.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:44.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:05:44.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509501 2024-09-17T19:05:44.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:44.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509501 2024-09-17T19:05:44.597 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509501 2024-09-17T19:05:44.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-09-17T19:05:44.598 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:05:44.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509502 -lt 64424509501 2024-09-17T19:05:44.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:44.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:44.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345973 2024-09-17T19:05:44.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:05:44.940 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345973 2024-09-17T19:05:44.940 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:44.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345973 2024-09-17T19:05:44.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345973' 2024-09-17T19:05:44.942 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345973 2024-09-17T19:05:44.943 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:05:45.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345973 -lt 85899345973 2024-09-17T19:05:45.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:45.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182446 2024-09-17T19:05:45.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:45.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:05:45.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182446 2024-09-17T19:05:45.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:45.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182446 2024-09-17T19:05:45.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-09-17T19:05:45.270 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.4 seq 107374182446 2024-09-17T19:05:45.270 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:05:45.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182447 -lt 107374182446 2024-09-17T19:05:45.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:45.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018917 2024-09-17T19:05:45.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:45.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:05:45.606 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018917 2024-09-17T19:05:45.606 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:45.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018917 2024-09-17T19:05:45.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018917' 2024-09-17T19:05:45.608 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.5 seq 128849018917 2024-09-17T19:05:45.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:05:45.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018918 -lt 128849018917 2024-09-17T19:05:45.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:45.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855395 2024-09-17T19:05:45.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:45.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:05:45.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855395 2024-09-17T19:05:45.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:45.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855395 2024-09-17T19:05:45.993 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.6 seq 150323855395 2024-09-17T19:05:45.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-09-17T19:05:45.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:05:46.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855395 -lt 150323855395 2024-09-17T19:05:46.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:46.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691862 2024-09-17T19:05:46.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:46.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:05:46.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:46.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691862 2024-09-17T19:05:46.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691862 2024-09-17T19:05:46.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-09-17T19:05:46.330 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.7 seq 171798691862 2024-09-17T19:05:46.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:05:46.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691863 -lt 171798691862 2024-09-17T19:05:46.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:46.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528333 2024-09-17T19:05:46.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:46.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:05:46.676 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528333 2024-09-17T19:05:46.676 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:46.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528333 2024-09-17T19:05:46.678 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.8 seq 193273528333 2024-09-17T19:05:46.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2024-09-17T19:05:46.679 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:05:47.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528333 -lt 193273528333 2024-09-17T19:05:47.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:47.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364814 2024-09-17T19:05:47.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:47.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:05:47.021 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364814 2024-09-17T19:05:47.021 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:47.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364814 2024-09-17T19:05:47.023 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.9 seq 214748364814 2024-09-17T19:05:47.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-09-17T19:05:47.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:05:47.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364814 -lt 214748364814 2024-09-17T19:05:47.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:05:47.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:05:47.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:05:47.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:05:47.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:05:47.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:05:47.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:05:47.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:05:47.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:05:47.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:05:47.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:05:48.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:05:48.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:05:48.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:05:48.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:05:48.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-17T19:05:48.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T19:05:48.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-17T19:05:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T19:05:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T19:05:49.901 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T19:05:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:05:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-17T19:05:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T19:05:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:05:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:05:50.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:05:50.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:05:50.930 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:05:50.930 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:05:50.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:05:50.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:05:50.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:05:51.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:05:51.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:05:51.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:05:51.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:05:51.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:05:51.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:05:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:05:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:05:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:05:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:05:51.418 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:51.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:05:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836559 2024-09-17T19:05:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836559 2024-09-17T19:05:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559' 2024-09-17T19:05:51.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:51.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:05:51.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673026 2024-09-17T19:05:51.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673026 2024-09-17T19:05:51.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026' 2024-09-17T19:05:51.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:51.854 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:05:52.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509504 2024-09-17T19:05:52.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509504 2024-09-17T19:05:52.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504' 2024-09-17T19:05:52.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:52.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:05:52.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345976 2024-09-17T19:05:52.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345976 2024-09-17T19:05:52.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976' 2024-09-17T19:05:52.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:52.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:05:52.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182449 2024-09-17T19:05:52.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182449 2024-09-17T19:05:52.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976 4-107374182449' 2024-09-17T19:05:52.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:52.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:05:52.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018920 2024-09-17T19:05:52.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018920 2024-09-17T19:05:52.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976 4-107374182449 5-128849018920' 2024-09-17T19:05:52.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:52.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:05:52.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855398 2024-09-17T19:05:52.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855398 2024-09-17T19:05:52.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976 4-107374182449 5-128849018920 6-150323855398' 2024-09-17T19:05:52.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:52.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:05:53.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-17T19:05:53.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-17T19:05:53.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976 4-107374182449 5-128849018920 6-150323855398 7-171798691865' 2024-09-17T19:05:53.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:53.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:05:53.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528336 2024-09-17T19:05:53.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528336 2024-09-17T19:05:53.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976 4-107374182449 5-128849018920 6-150323855398 7-171798691865 8-193273528336' 2024-09-17T19:05:53.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:05:53.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:05:53.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364817 2024-09-17T19:05:53.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364817 2024-09-17T19:05:53.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836559 1-42949673026 2-64424509504 3-85899345976 4-107374182449 5-128849018920 6-150323855398 7-171798691865 8-193273528336 9-214748364817' 2024-09-17T19:05:53.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:53.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836559 2024-09-17T19:05:53.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:53.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:05:53.297 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836559 2024-09-17T19:05:53.297 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:53.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836559 2024-09-17T19:05:53.298 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836559 2024-09-17T19:05:53.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-09-17T19:05:53.298 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:05:53.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836559 -lt 21474836559 2024-09-17T19:05:53.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:53.628 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673026 2024-09-17T19:05:53.628 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:53.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:05:53.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673026 2024-09-17T19:05:53.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:53.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673026 2024-09-17T19:05:53.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-09-17T19:05:53.633 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949673026 2024-09-17T19:05:53.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:05:54.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673027 -lt 42949673026 2024-09-17T19:05:54.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:54.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509504 2024-09-17T19:05:54.019 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:54.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:05:54.021 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509504 2024-09-17T19:05:54.022 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:54.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509504 2024-09-17T19:05:54.023 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509504 2024-09-17T19:05:54.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509504' 2024-09-17T19:05:54.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:05:54.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509505 -lt 64424509504 2024-09-17T19:05:54.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:54.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345976 2024-09-17T19:05:54.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:54.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:05:54.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345976 2024-09-17T19:05:54.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:54.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345976 2024-09-17T19:05:54.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345976' 2024-09-17T19:05:54.380 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345976 2024-09-17T19:05:54.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:05:54.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345976 -lt 85899345976 2024-09-17T19:05:54.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:54.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182449 2024-09-17T19:05:54.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:54.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:05:54.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182449 2024-09-17T19:05:54.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:54.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182449 2024-09-17T19:05:54.784 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.4 seq 107374182449 2024-09-17T19:05:54.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182449' 2024-09-17T19:05:54.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:05:55.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182450 -lt 107374182449 2024-09-17T19:05:55.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:55.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:55.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018920 2024-09-17T19:05:55.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:05:55.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018920 2024-09-17T19:05:55.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:55.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018920 2024-09-17T19:05:55.116 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.5 seq 128849018920 2024-09-17T19:05:55.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018920' 2024-09-17T19:05:55.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:05:55.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018920 -lt 128849018920 2024-09-17T19:05:55.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:55.508 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855398 2024-09-17T19:05:55.508 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:55.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:05:55.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855398 2024-09-17T19:05:55.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:55.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855398 2024-09-17T19:05:55.512 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.6 seq 150323855398 2024-09-17T19:05:55.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-09-17T19:05:55.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:05:55.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855398 -lt 150323855398 2024-09-17T19:05:55.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:55.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691865 2024-09-17T19:05:55.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:55.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:05:55.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691865 2024-09-17T19:05:55.859 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:55.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-17T19:05:55.860 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.7 seq 171798691865 2024-09-17T19:05:55.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-09-17T19:05:55.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:05:56.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691866 -lt 171798691865 2024-09-17T19:05:56.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:56.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528336 2024-09-17T19:05:56.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:56.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:05:56.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528336 2024-09-17T19:05:56.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:56.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528336 2024-09-17T19:05:56.185 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.8 seq 193273528336 2024-09-17T19:05:56.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-09-17T19:05:56.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:05:56.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528336 -lt 193273528336 2024-09-17T19:05:56.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:05:56.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364817 2024-09-17T19:05:56.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:05:56.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:05:56.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364817 2024-09-17T19:05:56.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:05:56.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364817 2024-09-17T19:05:56.526 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.9 seq 214748364817 2024-09-17T19:05:56.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-09-17T19:05:56.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:05:56.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364817 -lt 214748364817 2024-09-17T19:05:56.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:05:56.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:05:56.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:05:57.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:05:57.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:05:57.602 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:05:57.602 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:05:57.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:05:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T19:05:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:05:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:05:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:05:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:05:57.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:05:58.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:05:58.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:05:58.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:05:58.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:05:58.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:05:58.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:05:58.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:05:58.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:05:58.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:05:58.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:05:58.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:05:58.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:05:58.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:05:58.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:05:58.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:05:58.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T19:05:58.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:05:58.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:05:59.315 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:05:59.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:05:59.878 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:05:59.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:05:59.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:05:59.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T19:06:00.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-17T19:06:00.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T19:06:00.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T19:06:00.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:06:00.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T19:06:00.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-17T19:06:00.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-17T19:06:00.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T19:06:00.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T19:06:00.931 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-17T19:06:00.931 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:06:00.931 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr:9 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:01.268 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:01.269 INFO:tasks.workunit.client.0.smithi105.stderr:7' 2024-09-17T19:06:01.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-17T19:06:01.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-17T19:06:01.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-17T19:06:01.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-17T19:06:01.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-17T19:06:01.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:06:01.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:06:01.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:06:01.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:06:01.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:06:01.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:06:01.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:06:01.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:06:01.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T19:06:01.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:06:01.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:06:01.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T19:06:03.678 INFO:tasks.workunit.client.0.smithi105.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:06:04.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:06:04.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:06:04.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:06:04.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T19:06:04.230 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T19:06:04.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:06:04.230 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T19:06:04.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T19:06:04.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:06:04.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:06:04.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:06:04.264 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:04.262+0000 7f86fd9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:04.266 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:04.266+0000 7f86fd9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:04.268 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:04.266+0000 7f86fd9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:06:04.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:04.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:06:04.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:05.544 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:05.542+0000 7f86fd9437c0 -1 Falling back to public interface 2024-09-17T19:06:05.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:05.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:05.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:06:05.934 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:05.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:05.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:06:06.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:07.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:07.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:07.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:06:07.269 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:07.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:07.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:06:07.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:08.422 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:08.421+0000 7f86fd9437c0 -1 osd.3 68 log_to_monitors true 2024-09-17T19:06:08.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:08.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:08.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:06:08.598 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:06:08.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:08.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:06:09.043 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 72 up_thru 64 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/3381284912,v1:127.0.0.1:6827/3381284912] [v2:127.0.0.1:6828/3381284912,v1:127.0.0.1:6829/3381284912] exists,up 4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:06:09.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:06:09.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:06:09.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:06:09.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:06:09.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:06:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:06:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:06:09.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:06:09.045 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:06:09.045 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:06:09.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:06:09.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:06:09.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:06:09.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:06:09.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:06:09.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:06:09.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:06:09.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:06:09.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:06:09.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:06:09.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:06:09.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:09.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:06:09.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836564 2024-09-17T19:06:09.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836564 2024-09-17T19:06:09.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564' 2024-09-17T19:06:09.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:09.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:06:10.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673031 2024-09-17T19:06:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673031 2024-09-17T19:06:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031' 2024-09-17T19:06:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:10.168 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:06:10.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509509 2024-09-17T19:06:10.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509509 2024-09-17T19:06:10.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509' 2024-09-17T19:06:10.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:10.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:06:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645314 2024-09-17T19:06:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645314 2024-09-17T19:06:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314' 2024-09-17T19:06:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:10.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:06:10.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182454 2024-09-17T19:06:10.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182454 2024-09-17T19:06:10.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314 4-107374182454' 2024-09-17T19:06:10.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:10.636 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:06:10.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018925 2024-09-17T19:06:10.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018925 2024-09-17T19:06:10.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314 4-107374182454 5-128849018925' 2024-09-17T19:06:10.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:10.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:06:10.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855402 2024-09-17T19:06:10.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855402 2024-09-17T19:06:10.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314 4-107374182454 5-128849018925 6-150323855402' 2024-09-17T19:06:10.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:10.923 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:06:11.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691870 2024-09-17T19:06:11.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691870 2024-09-17T19:06:11.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314 4-107374182454 5-128849018925 6-150323855402 7-171798691870' 2024-09-17T19:06:11.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:11.064 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:06:11.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528340 2024-09-17T19:06:11.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528340 2024-09-17T19:06:11.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314 4-107374182454 5-128849018925 6-150323855402 7-171798691870 8-193273528340' 2024-09-17T19:06:11.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:11.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:06:11.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364822 2024-09-17T19:06:11.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364822 2024-09-17T19:06:11.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836564 1-42949673031 2-64424509509 3-309237645314 4-107374182454 5-128849018925 6-150323855402 7-171798691870 8-193273528340 9-214748364822' 2024-09-17T19:06:11.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:11.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836564 2024-09-17T19:06:11.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:11.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:06:11.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836564 2024-09-17T19:06:11.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:11.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836564 2024-09-17T19:06:11.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-09-17T19:06:11.341 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836564 2024-09-17T19:06:11.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:06:11.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836563 -lt 21474836564 2024-09-17T19:06:11.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:06:12.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:06:12.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:06:12.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836564 -lt 21474836564 2024-09-17T19:06:12.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:12.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673031 2024-09-17T19:06:12.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:12.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:06:12.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673031 2024-09-17T19:06:12.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:12.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673031 2024-09-17T19:06:12.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-17T19:06:12.994 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673031 2024-09-17T19:06:12.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:06:13.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673031 2024-09-17T19:06:13.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:13.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509509 2024-09-17T19:06:13.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:13.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:06:13.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509509 2024-09-17T19:06:13.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:13.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509509 2024-09-17T19:06:13.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-09-17T19:06:13.341 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509509 2024-09-17T19:06:13.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:06:13.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509509 -lt 64424509509 2024-09-17T19:06:13.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:13.680 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:13.680 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645314 2024-09-17T19:06:13.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:06:13.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645314 2024-09-17T19:06:13.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:13.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645314 2024-09-17T19:06:13.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-09-17T19:06:13.684 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 309237645314 2024-09-17T19:06:13.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:06:14.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645314 -lt 309237645314 2024-09-17T19:06:14.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:14.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182454 2024-09-17T19:06:14.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:14.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:06:14.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182454 2024-09-17T19:06:14.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:14.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182454 2024-09-17T19:06:14.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-09-17T19:06:14.070 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182454 2024-09-17T19:06:14.070 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:06:14.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182455 -lt 107374182454 2024-09-17T19:06:14.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:14.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018925 2024-09-17T19:06:14.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:14.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:06:14.405 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018925 2024-09-17T19:06:14.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:14.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018925 2024-09-17T19:06:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-09-17T19:06:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 128849018925 2024-09-17T19:06:14.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:06:14.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018925 -lt 128849018925 2024-09-17T19:06:14.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:14.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855402 2024-09-17T19:06:14.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:14.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:06:14.746 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:14.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855402 2024-09-17T19:06:14.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855402 2024-09-17T19:06:14.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-09-17T19:06:14.748 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855402 2024-09-17T19:06:14.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:06:15.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855403 -lt 150323855402 2024-09-17T19:06:15.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:15.106 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:15.106 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691870 2024-09-17T19:06:15.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:06:15.108 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691870 2024-09-17T19:06:15.108 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:15.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691870 2024-09-17T19:06:15.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-09-17T19:06:15.110 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691870 2024-09-17T19:06:15.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:06:15.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691870 -lt 171798691870 2024-09-17T19:06:15.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:15.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528340 2024-09-17T19:06:15.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:15.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:06:15.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528340 2024-09-17T19:06:15.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:15.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528340 2024-09-17T19:06:15.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-09-17T19:06:15.473 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528340 2024-09-17T19:06:15.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:06:15.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528341 -lt 193273528340 2024-09-17T19:06:15.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:15.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364822 2024-09-17T19:06:15.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:15.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:06:15.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364822 2024-09-17T19:06:15.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:15.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364822 2024-09-17T19:06:15.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364822' 2024-09-17T19:06:15.822 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 214748364822 2024-09-17T19:06:15.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:06:16.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364822 -lt 214748364822 2024-09-17T19:06:16.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:06:16.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:06:16.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:06:16.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:06:16.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:06:16.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:06:16.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:06:16.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:06:16.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:06:16.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:06:16.524 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:06:16.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:06:16.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:06:16.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:06:16.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:06:17.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:06:17.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:06:17.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:06:17.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T19:06:17.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:06:17.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:06:17.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:06:17.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:06:17.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:17.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:17.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:05:49.741887+0000 2024-09-17T19:06:17.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:06:18.256 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:06:18.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:05:49.741887+0000 2024-09-17T19:06:18.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:06:18.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:05:49.741887+0000 2024-09-17T19:06:18.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:06:18.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:06:18.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:18.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:18.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:18.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:18.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:18.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:18.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:05:49.741887+0000 '>' 2024-09-17T19:05:49.741887+0000 2024-09-17T19:06:18.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:06:19.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:06:19.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:19.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:19.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:19.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:19.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:19.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:19.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:18.515907+0000 '>' 2024-09-17T19:05:49.741887+0000 2024-09-17T19:06:19.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:06:19.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:06:21.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:06:21.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:06:21.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:06:21.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:06:21.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:06:21.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:06:21.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:06:21.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:06:21.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:06:22.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:06:22.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:06:22.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:06:22.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:06:22.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:06:22.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:06:22.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:06:22.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-17T19:06:22.820 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T19:06:22.820 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T19:06:22.820 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T19:06:23.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:06:23.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:06:23.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:06:23.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:06:23.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:06:23.116 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:06:23.116 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:06:23.116 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:06:23.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:06:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:06:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:06:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:06:23.118 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.9 2024-09-17T19:06:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:06:23.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:06:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:06:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:06:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:06:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:06:23.153 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:23.153+0000 7fd92ec4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:23.155 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:23.153+0000 7fd92ec4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:23.157 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:23.157+0000 7fd92ec4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:23.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:23.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:06:24.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:24.680 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:24.677+0000 7fd92ec4b7c0 -1 Falling back to public interface 2024-09-17T19:06:25.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:25.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:25.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:06:25.028 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:25.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:25.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:06:25.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:26.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:26.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:26.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:06:26.367 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:26.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:26.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:06:26.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:06:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:06:27.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:27.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:06:28.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:28.078 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:28.077+0000 7fd92ec4b7c0 -1 osd.9 73 log_to_monitors true 2024-09-17T19:06:29.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:29.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:29.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:06:29.036 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:06:29.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:29.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/3318225817,v1:127.0.0.1:6875/3318225817] [v2:127.0.0.1:6876/3318225817,v1:127.0.0.1:6877/3318225817] exists,up 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:06:29.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:06:29.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:06:29.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:06:29.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:06:29.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:06:29.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:06:29.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:06:29.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:06:29.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:06:29.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:06:29.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:06:29.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:06:29.722 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:06:30.221 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:06:30.222 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:06:30.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:06:30.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:30.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:06:30.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-17T19:06:30.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-17T19:06:30.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-17T19:06:30.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:30.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:06:30.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673036 2024-09-17T19:06:30.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673036 2024-09-17T19:06:30.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036' 2024-09-17T19:06:30.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:30.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:06:30.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509514 2024-09-17T19:06:30.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509514 2024-09-17T19:06:30.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514' 2024-09-17T19:06:30.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:30.946 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:06:31.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645319 2024-09-17T19:06:31.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645319 2024-09-17T19:06:31.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319' 2024-09-17T19:06:31.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:31.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:06:31.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182459 2024-09-17T19:06:31.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182459 2024-09-17T19:06:31.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319 4-107374182459' 2024-09-17T19:06:31.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:31.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:06:31.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018930 2024-09-17T19:06:31.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018930 2024-09-17T19:06:31.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319 4-107374182459 5-128849018930' 2024-09-17T19:06:31.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:31.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:06:31.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855407 2024-09-17T19:06:31.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855407 2024-09-17T19:06:31.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319 4-107374182459 5-128849018930 6-150323855407' 2024-09-17T19:06:31.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:31.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:06:31.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691875 2024-09-17T19:06:31.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691875 2024-09-17T19:06:31.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319 4-107374182459 5-128849018930 6-150323855407 7-171798691875' 2024-09-17T19:06:31.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:31.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:06:31.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528345 2024-09-17T19:06:31.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528345 2024-09-17T19:06:31.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319 4-107374182459 5-128849018930 6-150323855407 7-171798691875 8-193273528345' 2024-09-17T19:06:31.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:31.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:06:31.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481794 2024-09-17T19:06:31.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481794 2024-09-17T19:06:31.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673036 2-64424509514 3-309237645319 4-107374182459 5-128849018930 6-150323855407 7-171798691875 8-193273528345 9-330712481794' 2024-09-17T19:06:31.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:31.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-17T19:06:31.944 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:31.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:06:31.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-17T19:06:31.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:31.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-17T19:06:31.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-17T19:06:31.946 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836569 2024-09-17T19:06:31.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:06:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-17T19:06:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:32.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673036 2024-09-17T19:06:32.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:32.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:06:32.250 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673036 2024-09-17T19:06:32.251 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:32.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673036 2024-09-17T19:06:32.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-09-17T19:06:32.252 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673036 2024-09-17T19:06:32.252 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:06:32.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673036 2024-09-17T19:06:32.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:32.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509514 2024-09-17T19:06:32.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:32.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:06:32.612 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509514 2024-09-17T19:06:32.612 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:32.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509514 2024-09-17T19:06:32.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-09-17T19:06:32.614 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509514 2024-09-17T19:06:32.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:06:32.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509514 -lt 64424509514 2024-09-17T19:06:32.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:32.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645319 2024-09-17T19:06:32.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:32.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:06:32.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645319 2024-09-17T19:06:32.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645319 2024-09-17T19:06:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645319' 2024-09-17T19:06:32.960 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 309237645319 2024-09-17T19:06:32.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:06:33.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645319 -lt 309237645319 2024-09-17T19:06:33.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:33.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182459 2024-09-17T19:06:33.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:33.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:06:33.288 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182459 2024-09-17T19:06:33.288 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:33.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182459 2024-09-17T19:06:33.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-09-17T19:06:33.290 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182459 2024-09-17T19:06:33.290 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:06:33.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182459 -lt 107374182459 2024-09-17T19:06:33.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018930 2024-09-17T19:06:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:06:33.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018930 2024-09-17T19:06:33.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:33.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018930 2024-09-17T19:06:33.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-09-17T19:06:33.641 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 128849018930 2024-09-17T19:06:33.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:06:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018930 -lt 128849018930 2024-09-17T19:06:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:34.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855407 2024-09-17T19:06:34.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:34.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:06:34.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855407 2024-09-17T19:06:34.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855407 2024-09-17T19:06:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-09-17T19:06:34.047 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855407 2024-09-17T19:06:34.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:06:34.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855408 -lt 150323855407 2024-09-17T19:06:34.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:34.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691875 2024-09-17T19:06:34.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:34.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:06:34.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691875 2024-09-17T19:06:34.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691875 2024-09-17T19:06:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-09-17T19:06:34.376 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691875 2024-09-17T19:06:34.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:06:34.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691875 -lt 171798691875 2024-09-17T19:06:34.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:34.713 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528345 2024-09-17T19:06:34.713 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:34.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:06:34.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528345 2024-09-17T19:06:34.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528345 2024-09-17T19:06:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-09-17T19:06:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528345 2024-09-17T19:06:34.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:06:35.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528346 -lt 193273528345 2024-09-17T19:06:35.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:35.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-330712481794 2024-09-17T19:06:35.075 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:35.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:06:35.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-330712481794 2024-09-17T19:06:35.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:35.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481794 2024-09-17T19:06:35.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-09-17T19:06:35.080 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 330712481794 2024-09-17T19:06:35.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:06:35.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481795 -lt 330712481794 2024-09-17T19:06:35.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:06:35.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:06:35.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:06:35.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:06:35.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:06:35.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:06:35.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:06:35.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:06:35.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:06:35.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:06:35.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:06:36.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:06:36.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:06:36.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:06:36.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:06:36.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:06:36.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:06:36.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:06:36.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:06:36.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:06:36.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-17T19:06:36.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T19:06:36.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T19:06:36.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-17T19:06:36.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:06:36.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:06:36.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:06:36.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:06:36.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:06:36.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-17T19:06:39.113 INFO:tasks.workunit.client.0.smithi105.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:06:39.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:06:39.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:06:39.655 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:06:39.655 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:06:39.655 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:06:39.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:06:39.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:06:39.657 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.5 2024-09-17T19:06:39.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:06:39.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:06:39.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:06:39.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:06:39.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:06:39.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:06:39.697 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:39.697+0000 7f9dc89d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:39.699 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:39.697+0000 7f9dc89d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:39.701 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:39.701+0000 7f9dc89d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:06:39.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:06:39.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:06:39.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:06:39.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:06:40.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:06:40.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:40.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:06:40.000 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:06:40.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:40.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:06:40.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:41.004 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:41.001+0000 7f9dc89d87c0 -1 Falling back to public interface 2024-09-17T19:06:41.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:41.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:41.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:06:41.348 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:41.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:41.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:06:41.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:42.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:42.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:42.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:06:42.677 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:42.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:42.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:06:43.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:43.858 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:06:43.837+0000 7f9dc89d87c0 -1 osd.5 79 log_to_monitors true 2024-09-17T19:06:44.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:44.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:44.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:06:44.015 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:06:44.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:44.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:06:44.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:06:45.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:06:45.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:06:45.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:06:45.363 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:06:45.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:06:45.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:06:45.793 INFO:tasks.workunit.client.0.smithi105.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/1385310975,v1:127.0.0.1:6843/1385310975] [v2:127.0.0.1:6844/1385310975,v1:127.0.0.1:6845/1385310975] exists,up ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:06:45.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:06:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:06:45.795 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:06:45.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:06:45.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:06:45.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:06:45.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:06:45.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:06:45.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:06:45.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:06:45.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:06:45.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:06:46.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:46.450 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:06:46.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836573 2024-09-17T19:06:46.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836573 2024-09-17T19:06:46.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573' 2024-09-17T19:06:46.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:46.690 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:06:46.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673040 2024-09-17T19:06:46.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673040 2024-09-17T19:06:46.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040' 2024-09-17T19:06:46.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:46.853 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:06:46.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509518 2024-09-17T19:06:46.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509518 2024-09-17T19:06:46.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518' 2024-09-17T19:06:46.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:46.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:06:47.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645323 2024-09-17T19:06:47.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645323 2024-09-17T19:06:47.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323' 2024-09-17T19:06:47.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:47.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:06:47.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182463 2024-09-17T19:06:47.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182463 2024-09-17T19:06:47.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323 4-107374182463' 2024-09-17T19:06:47.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:47.288 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:06:47.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-17T19:06:47.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-17T19:06:47.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323 4-107374182463 5-360777252866' 2024-09-17T19:06:47.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:47.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:06:47.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855411 2024-09-17T19:06:47.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855411 2024-09-17T19:06:47.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323 4-107374182463 5-360777252866 6-150323855411' 2024-09-17T19:06:47.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:47.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:06:47.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-17T19:06:47.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-17T19:06:47.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323 4-107374182463 5-360777252866 6-150323855411 7-171798691879' 2024-09-17T19:06:47.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:47.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:06:47.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528350 2024-09-17T19:06:47.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528350 2024-09-17T19:06:47.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323 4-107374182463 5-360777252866 6-150323855411 7-171798691879 8-193273528350' 2024-09-17T19:06:47.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:06:47.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:06:48.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=330712481798 2024-09-17T19:06:48.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 330712481798 2024-09-17T19:06:48.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836573 1-42949673040 2-64424509518 3-309237645323 4-107374182463 5-360777252866 6-150323855411 7-171798691879 8-193273528350 9-330712481798' 2024-09-17T19:06:48.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:48.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836573 2024-09-17T19:06:48.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:48.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:06:48.021 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836573 2024-09-17T19:06:48.021 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:48.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836573 2024-09-17T19:06:48.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-17T19:06:48.022 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836573 2024-09-17T19:06:48.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:06:48.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836573 -lt 21474836573 2024-09-17T19:06:48.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:48.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673040 2024-09-17T19:06:48.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:48.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:06:48.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673040 2024-09-17T19:06:48.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:48.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673040 2024-09-17T19:06:48.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-09-17T19:06:48.347 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673040 2024-09-17T19:06:48.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:06:48.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673041 -lt 42949673040 2024-09-17T19:06:48.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:48.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:48.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509518 2024-09-17T19:06:48.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:06:48.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509518 2024-09-17T19:06:48.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:48.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509518 2024-09-17T19:06:48.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509518' 2024-09-17T19:06:48.682 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509518 2024-09-17T19:06:48.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:06:49.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509518 -lt 64424509518 2024-09-17T19:06:49.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:49.034 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645323 2024-09-17T19:06:49.034 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:49.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:06:49.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645323 2024-09-17T19:06:49.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:49.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645323 2024-09-17T19:06:49.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645323' 2024-09-17T19:06:49.037 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 309237645323 2024-09-17T19:06:49.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:06:49.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645323 -lt 309237645323 2024-09-17T19:06:49.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:49.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182463 2024-09-17T19:06:49.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:49.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:06:49.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182463 2024-09-17T19:06:49.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:49.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182463 2024-09-17T19:06:49.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-09-17T19:06:49.379 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182463 2024-09-17T19:06:49.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:06:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182463 -lt 107374182463 2024-09-17T19:06:49.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:49.750 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252866 2024-09-17T19:06:49.750 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:49.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:06:49.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252866 2024-09-17T19:06:49.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-17T19:06:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-09-17T19:06:49.753 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 360777252866 2024-09-17T19:06:49.754 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:06:50.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252867 -lt 360777252866 2024-09-17T19:06:50.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:50.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:50.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855411 2024-09-17T19:06:50.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:06:50.140 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855411 2024-09-17T19:06:50.140 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:50.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855411 2024-09-17T19:06:50.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-09-17T19:06:50.142 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855411 2024-09-17T19:06:50.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:06:50.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855412 -lt 150323855411 2024-09-17T19:06:50.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:50.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-17T19:06:50.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:50.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:06:50.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-17T19:06:50.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:50.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-17T19:06:50.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-17T19:06:50.495 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691879 2024-09-17T19:06:50.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:06:50.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691879 -lt 171798691879 2024-09-17T19:06:50.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:50.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528350 2024-09-17T19:06:50.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:50.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:06:50.853 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528350 2024-09-17T19:06:50.853 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:50.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528350 2024-09-17T19:06:50.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-09-17T19:06:50.854 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528350 2024-09-17T19:06:50.855 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:06:51.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528350 -lt 193273528350 2024-09-17T19:06:51.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:06:51.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:06:51.217 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-330712481798 2024-09-17T19:06:51.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:06:51.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-330712481798 2024-09-17T19:06:51.219 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:06:51.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=330712481798 2024-09-17T19:06:51.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-09-17T19:06:51.221 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 330712481798 2024-09-17T19:06:51.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:06:51.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 330712481799 -lt 330712481798 2024-09-17T19:06:51.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:06:51.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:06:51.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:06:51.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:06:51.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:06:51.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:06:51.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:06:51.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:06:51.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:06:51.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:06:51.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:06:52.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:06:52.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:06:52.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:06:52.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:06:52.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:06:52.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:06:52.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:06:52.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T19:06:52.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:06:52.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:06:52.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:06:52.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:06:53.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T19:06:53.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T19:06:53.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:06:53.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:06:53.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:53.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:53.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:53.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:53.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:53.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:06:53.662 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:06:53.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:53.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:06:53.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:53.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:06:53.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:06:53.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:53.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:53.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:53.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:53.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:53.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:53.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:18.515907+0000 '>' 2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:53.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:06:54.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:06:54.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:54.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:54.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:54.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:54.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:54.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:55.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:18.515907+0000 '>' 2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:55.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:06:56.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:06:56.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:56.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:56.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:56.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:56.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:56.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:56.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:18.515907+0000 '>' 2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:06:57.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:06:57.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:57.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:57.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:57.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:57.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:57.658 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:57.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:18.515907+0000 '>' 2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:57.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:06:58.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:06:58.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:06:58.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:06:58.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:06:58.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:06:58.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:06:58.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:06:59.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:18.515907+0000 '>' 2024-09-17T19:06:18.515907+0000 2024-09-17T19:06:59.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:07:00.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:07:00.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:07:00.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:07:00.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:00.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:00.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:00.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:00.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:54.482721+0000 '>' 2024-09-17T19:06:18.515907+0000 2024-09-17T19:07:00.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:07:00.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:07:02.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:07:02.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:07:02.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:07:02.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:07:02.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:07:02.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:07:02.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:07:02.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:07:02.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:07:02.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:07:02.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-17T19:07:03.684 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T19:07:03.684 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T19:07:03.684 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T19:07:03.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:07:03.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:07:03.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:07:03.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:07:03.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:07:03.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:07:03.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:07:03.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:07:03.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:07:03.996 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.9 2024-09-17T19:07:03.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:07:03.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:07:03.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:07:03.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:07:03.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:07:03.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:07:04.031 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:07:04.028+0000 7f0d1f8447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:04.033 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:07:04.032+0000 7f0d1f8447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:04.035 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:07:04.032+0000 7f0d1f8447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:04.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:07:04.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:07:04.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:07:04.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:07:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:07:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:07:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:07:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:04.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:05.348 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:07:05.344+0000 7f0d1f8447c0 -1 Falling back to public interface 2024-09-17T19:07:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:07:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:07:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:06.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:07.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:07.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:07.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:07:07.044 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:07:07.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:07.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:07.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:08.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:08.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:08.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:07:08.366 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:07:08.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:08.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:08.652 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:07:08.648+0000 7f0d1f8447c0 -1 osd.9 86 log_to_monitors true 2024-09-17T19:07:08.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:09.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:09.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:09.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:07:09.709 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:07:09.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:09.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [77,86) [v2:127.0.0.1:6874/2498419518,v1:127.0.0.1:6875/2498419518] [v2:127.0.0.1:6876/2498419518,v1:127.0.0.1:6877/2498419518] exists,up 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:07:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:07:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:07:10.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:07:10.169 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:07:10.170 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:07:10.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:07:10.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:07:10.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:07:10.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:07:10.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:07:10.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:07:10.699 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:07:10.700 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:07:10.700 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:07:10.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:07:10.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:10.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:07:10.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-17T19:07:10.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-17T19:07:10.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-17T19:07:10.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:10.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:07:11.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673046 2024-09-17T19:07:11.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673046 2024-09-17T19:07:11.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046' 2024-09-17T19:07:11.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:11.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:07:11.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509524 2024-09-17T19:07:11.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509524 2024-09-17T19:07:11.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524' 2024-09-17T19:07:11.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:11.270 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:07:11.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645329 2024-09-17T19:07:11.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645329 2024-09-17T19:07:11.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329' 2024-09-17T19:07:11.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:11.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:07:11.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182469 2024-09-17T19:07:11.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182469 2024-09-17T19:07:11.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329 4-107374182469' 2024-09-17T19:07:11.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:11.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:07:11.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252872 2024-09-17T19:07:11.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252872 2024-09-17T19:07:11.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329 4-107374182469 5-360777252872' 2024-09-17T19:07:11.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:11.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:07:11.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855417 2024-09-17T19:07:11.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855417 2024-09-17T19:07:11.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329 4-107374182469 5-360777252872 6-150323855417' 2024-09-17T19:07:11.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:11.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:07:12.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691885 2024-09-17T19:07:12.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691885 2024-09-17T19:07:12.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329 4-107374182469 5-360777252872 6-150323855417 7-171798691885' 2024-09-17T19:07:12.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:12.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:07:12.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528355 2024-09-17T19:07:12.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528355 2024-09-17T19:07:12.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329 4-107374182469 5-360777252872 6-150323855417 7-171798691885 8-193273528355' 2024-09-17T19:07:12.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:12.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:07:12.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-17T19:07:12.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-17T19:07:12.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673046 2-64424509524 3-309237645329 4-107374182469 5-360777252872 6-150323855417 7-171798691885 8-193273528355 9-386547056642' 2024-09-17T19:07:12.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:12.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-17T19:07:12.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:12.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:07:12.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-17T19:07:12.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:12.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-17T19:07:12.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-17T19:07:12.324 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836579 2024-09-17T19:07:12.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:07:12.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-17T19:07:12.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:12.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673046 2024-09-17T19:07:12.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:12.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:07:12.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673046 2024-09-17T19:07:12.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:12.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673046 2024-09-17T19:07:12.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-09-17T19:07:12.642 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673046 2024-09-17T19:07:12.643 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:07:12.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673046 -lt 42949673046 2024-09-17T19:07:12.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:12.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509524 2024-09-17T19:07:12.994 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:12.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:07:12.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509524 2024-09-17T19:07:12.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:12.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509524 2024-09-17T19:07:12.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-17T19:07:12.998 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509524 2024-09-17T19:07:12.999 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:07:13.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-17T19:07:13.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:13.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645329 2024-09-17T19:07:13.351 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:07:13.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645329 2024-09-17T19:07:13.353 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:13.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645329 2024-09-17T19:07:13.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645329' 2024-09-17T19:07:13.355 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 309237645329 2024-09-17T19:07:13.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:07:13.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645329 -lt 309237645329 2024-09-17T19:07:13.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:13.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182469 2024-09-17T19:07:13.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:13.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:07:13.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182469 2024-09-17T19:07:13.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:13.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182469 2024-09-17T19:07:13.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-09-17T19:07:13.721 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182469 2024-09-17T19:07:13.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:07:14.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182470 -lt 107374182469 2024-09-17T19:07:14.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:14.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:14.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-360777252872 2024-09-17T19:07:14.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:07:14.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-360777252872 2024-09-17T19:07:14.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:14.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252872 2024-09-17T19:07:14.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 360777252872' 2024-09-17T19:07:14.107 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 360777252872 2024-09-17T19:07:14.107 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:07:14.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252872 -lt 360777252872 2024-09-17T19:07:14.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:14.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855417 2024-09-17T19:07:14.436 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:14.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:07:14.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855417 2024-09-17T19:07:14.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:14.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855417 2024-09-17T19:07:14.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-09-17T19:07:14.440 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855417 2024-09-17T19:07:14.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:07:14.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855418 -lt 150323855417 2024-09-17T19:07:14.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:14.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691885 2024-09-17T19:07:14.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:14.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:07:14.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691885 2024-09-17T19:07:14.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:14.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691885 2024-09-17T19:07:14.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-09-17T19:07:14.783 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691885 2024-09-17T19:07:14.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:07:15.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691885 -lt 171798691885 2024-09-17T19:07:15.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:15.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528355 2024-09-17T19:07:15.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:15.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:07:15.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528355 2024-09-17T19:07:15.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:15.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528355 2024-09-17T19:07:15.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-09-17T19:07:15.130 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528355 2024-09-17T19:07:15.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:07:15.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528356 -lt 193273528355 2024-09-17T19:07:15.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:15.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-386547056642 2024-09-17T19:07:15.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:15.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:07:15.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-386547056642 2024-09-17T19:07:15.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:15.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-17T19:07:15.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-09-17T19:07:15.464 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 386547056642 2024-09-17T19:07:15.464 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:07:15.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056643 -lt 386547056642 2024-09-17T19:07:15.816 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:07:15.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:15.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:16.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:07:16.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:16.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:16.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:16.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:16.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:16.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:16.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:16.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:07:16.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:16.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:16.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:16.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:07:16.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:07:16.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:07:16.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:07:17.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:07:17.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-17T19:07:17.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T19:07:17.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T19:07:17.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 204750"' 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 204750' 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 204752"' 2024-09-17T19:07:17.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 204752' 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/204753: /' 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 204750 204752' 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 204750 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:07:17.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/204755: /' 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:07:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:07:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:07:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:07:20.753 INFO:tasks.workunit.client.0.smithi105.stderr:204753: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:07:20.754 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:07:20.755 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:07:20.755 INFO:tasks.workunit.client.0.smithi105.stderr:204753: start osd.5 2024-09-17T19:07:20.755 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:07:20.873 INFO:tasks.workunit.client.0.smithi105.stderr:2204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.873 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:07:20.873 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:07:20.873 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:07:20.873 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.874 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:07:20.875 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:07:20.877 INFO:tasks.workunit.client.0.smithi105.stderr:204755: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:07:20.878 INFO:tasks.workunit.client.0.smithi105.stderr:204755: start osd.9 2024-09-17T19:07:20.879 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:07:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:07:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:07:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:07:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:07:25.437 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2024-09-17T19:07:20.788+0000 7f33ee9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2024-09-17T19:07:20.792+0000 7f33ee9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2024-09-17T19:07:20.792+0000 7f33ee9fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:07:25.438 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 0 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:25.439 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2024-09-17T19:07:22.100+0000 7f33ee9fb7c0 -1 Falling back to public interface 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 1 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:25.440 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:25.441 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:25.441 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:25.441 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:07:25.441 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2 2024-09-17T19:07:25.441 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:25.441 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 3 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:25.442 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2024-09-17T19:07:25.268+0000 7f33ee9fb7c0 -1 osd.5 92 log_to_monitors true 2024-09-17T19:07:26.151 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers04755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:07:26.151 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:07:26.151 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:07:26.151 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2024-09-17T19:07:20.904+0000 7f65f88d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2024-09-17T19:07:20.904+0000 7f65f88d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2024-09-17T19:07:20.908+0000 7f65f88d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:07:26.152 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 0 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2024-09-17T19:07:22.436+0000 7f65f88d77c0 -1 Falling back to public interface 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:07:26.153 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 1 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 3 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:28.124 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2024-09-17T19:07:26.148+0000 7f65f88d77c0 -.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 4 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:28.125 INFO:tasks.workunit.client.0.smithi105.stderr:204753: osd.5 up in weight 1 up_from 96 up_thru 84 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6842/1131932918,v1:127.0.0.1:6843/1131932918] [v2:127.0.0.1:6844/1131932918,v1:127.0.0.1:6845/1131932918] exists,up ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:07:28.126 INFO:tasks.workunit.client.0.smithi105.stderr:204753: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:07:28.127 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 1 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 2 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 3 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 4 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 5 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 6 2024-09-17T19:07:28.128 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 7 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 8 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: 9' 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836583 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836583 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583' 2024-09-17T19:07:28.129 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /1 osd.9 92 log_to_monitors true 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 4 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:28.973 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 5 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:28.974 INFO:tasks.workunit.client.0.smithi105.stderr:204755: osd.9 up in weight 1 up_from 98 up_thru 0 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6874/4045416216,v1:127.0.0.1:6875/4045416216] [v2:127.0.0.1:6876/4045416216,v1:127.0.0.1:6877/4045416216] exists,up 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:07:28.975 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:07:28.976 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 1 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 2 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 3 2024-09-17T19:07:28.977 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 4 2024-09-17T19:07:28.978 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 5 2024-09-17T19:07:28.978 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 6 2024-09-17T19:07:28.978 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 7 2024-09-17T19:07:28.978 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 8 2024-09-17T19:07:28.978 INFO:tasks.workunit.client.0.smithi105.stderr:204755: 9' 2024-09-17T19:07:28.978 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:07:29.803 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:07:29.803 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673051 2024-09-17T19:07:29.803 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673051 2024-09-17T19:07:29.803 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051' 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509529 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509529 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529' 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645334 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645334 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334' 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:07:29.804 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182474 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182474 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334 4-107374182474' 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860418 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860418 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334 4-107374182474 5-412316860418' 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855422 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855422 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334 4-107374182474 5-412316860418 6-150323855422' 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:29.805 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:07:30.375 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.375 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836584 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836584 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584' 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673052 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673052 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052' 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.376 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509530 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509530 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530' 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645335 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645335 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335' 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182475 2024-09-17T19:07:30.377 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182475 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335 4-107374182475' 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=412316860419 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 412316860419 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335 4-107374182475 5-412316860419' 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:07:30.378 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855423 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /hpg_stats: seq=171798691889 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691889 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334 4-107374182474 5-412316860418 6-150323855422 7-171798691889' 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528360 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528360 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334 4-107374182474 5-412316860418 6-150323855422 7-171798691889 8-193273528360' 2024-09-17T19:07:30.561 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836583 1-42949673051 2-64424509529 3-309237645334 4-107374182474 5-412316860418 6-150323855422 7-171798691889 8-193273528360 9-420906795010' 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836583 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836583 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836583 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.0 seq 21474836583 2024-09-17T19:07:30.562 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836583 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673051 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673051 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673051 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-09-17T19:07:30.563 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.1 seq 42949673051 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_some/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855423 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335 4-107374182475 5-412316860419 6-150323855423' 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691891 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691891 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335 4-107374182475 5-412316860419 6-150323855423 7-171798691891' 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528361 2024-09-17T19:07:31.107 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528361 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335 4-107374182475 5-412316860419 6-150323855423 7-171798691891 8-193273528361' 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795011 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795011 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836584 1-42949673052 2-64424509530 3-309237645335 4-107374182475 5-412316860419 6-150323855423 7-171798691891 8-193273528361 9-420906795011' 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836584 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:31.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836584 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836584 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.0 seq 21474836584 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836584 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673052 2024-09-17T19:07:31.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:31.993 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hetats: ceph osd last-stat-seq 1 2024-09-17T19:07:31.993 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673052 -lt 42949673051 2024-09-17T19:07:31.993 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:31.993 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509529 2024-09-17T19:07:31.993 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:31.993 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509529 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509529 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.2 seq 64424509529 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509529 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:31.994 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645334 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645334 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645334 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645334' 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.3 seq 309237645334 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645334 -lt 309237645334 2024-09-17T19:07:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182474 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182474 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182474 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.4 seq 107374182474 2024-09-17T19:07:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:07:31.997 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182475 -lt 107374182474 2024-09-17T19:07:31.997 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:31.997 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-412316860418 2024-09-17T19:07:32.108 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/celpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:07:32.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673052 2024-09-17T19:07:32.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:32.108 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673052 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.1 seq 42949673052 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673052 -lt 42949673052 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509530 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:07:32.109 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509530 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509530 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.2 seq 64424509530 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509530 -lt 64424509530 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645335 2024-09-17T19:07:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645335 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645335 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645335' 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.3 seq 309237645335 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645335 -lt 309237645335 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:32.111 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182475 2024-09-17T19:07:32.112 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:32.112 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:07:32.112 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182475 2024-09-17T19:07:32.112 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:32.112 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182475 2024-09-17T19:07:33.034 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.cliephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.034 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:07:33.034 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-412316860418 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860418 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 412316860418' 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.5 seq 412316860418 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860418 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855422 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855422 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855422 2024-09-17T19:07:33.035 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.6 seq 150323855422 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855423 -lt 150323855422 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691889 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691889 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691889 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.7 seq 171798691889 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:07:33.036 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691891 -lt 171798691889 2024-09-17T19:07:33.037 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:33.037 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528360 2024-09-17T19:07:33.037 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.037 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:07:33.037 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528360 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cnt.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.4 seq 107374182475 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182475 -lt 107374182475 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-412316860419 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:07:33.370 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-412316860419 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=412316860419 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.5 seq 412316860419 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 412316860420 -lt 412316860419 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855423 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855423 2024-09-17T19:07:33.371 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855423 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.6 seq 150323855423 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855423 -lt 150323855423 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691891 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691891 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:33.372 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691891 2024-09-17T19:07:33.373 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-09-17T19:07:33.373 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.7 seq 171798691891 2024-09-17T19:07:33.373 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:07:33.373 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691891 -lt 171798691891 2024-09-17T19:07:34.784 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntuut -d - -f 2 2024-09-17T19:07:34.784 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528360 2024-09-17T19:07:34.784 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-09-17T19:07:34.784 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.8 seq 193273528360 2024-09-17T19:07:34.784 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:07:34.784 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528361 -lt 193273528360 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-420906795010 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-420906795010 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 420906795010' 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: waiting osd.9 seq 420906795010 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795011 -lt 420906795010 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:34.785 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T19:07:34.786 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:07:34.787 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cepht/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528361 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528361 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528361 2024-09-17T19:07:34.826 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.8 seq 193273528361 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528361 -lt 193273528361 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-420906795011 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-420906795011 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795011 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 420906795011' 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: waiting osd.9 seq 420906795011 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:07:34.827 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795011 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:34.828 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/staest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:36.286 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:07:36.287 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:07:36.288 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:07:36.288 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:07:36.288 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:07:36.288 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:07:36.288 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:36.443 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qndalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:36.443 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:36.443 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:36.444 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:07:36.445 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:07:36.969 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:07:36.970 INFO:tasks.workunit.client.0.smithi105.stderr:204753: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:07:36.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:07:36.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:07:36.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 204752 2024-09-17T19:07:37.331 INFO:tasks.workunit.client.0.smithi105.stderr:ers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:07:37.331 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:07:37.332 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:07:37.333 INFO:tasks.workunit.client.0.smithi105.stderr:204755: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:07:37.333 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:07:37.333 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:07:37.333 INFO:tasks.workunit.client.0.smithi105.stderr:204755: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:07:37.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:07:37.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:07:37.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T19:07:37.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T19:07:37.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:07:37.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:07:37.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:37.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:37.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:37.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:38.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:38.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:07:38.351 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:07:38.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:38.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:07:38.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:38.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:07:38.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:07:38.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:07:38.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:07:38.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:38.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:38.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:38.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:38.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:54.482721+0000 '>' 2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:38.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:07:39.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:07:39.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:07:39.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:07:39.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:39.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:39.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:39.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:54.482721+0000 '>' 2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:07:41.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:07:41.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:07:41.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:07:41.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:41.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:41.054 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:41.054 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:41.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:54.482721+0000 '>' 2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:41.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:07:42.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:07:42.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:07:42.391 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:07:42.391 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:42.391 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:42.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:42.392 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:42.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:06:54.482721+0000 '>' 2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:42.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:07:43.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:07:43.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:07:43.727 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:07:43.728 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:07:43.728 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:07:43.728 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:07:43.728 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:07:44.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:07:39.225512+0000 '>' 2024-09-17T19:06:54.482721+0000 2024-09-17T19:07:44.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:07:44.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:07:46.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 208493"' 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 208493' 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:07:46.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:07:46.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 208495"' 2024-09-17T19:07:46.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 208495' 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 208493 208495' 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 208493 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/208496: /' 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:46.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/208498: /' 2024-09-17T19:07:47.200 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:07:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:07:47.202 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-17T19:07:47.203 INFO:tasks.workunit.client.0.smithi105.stderr:208496: _ 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: hinfo_key 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: snapset 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:07:47.204 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:07:47.205 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:07:47.205 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:07:47.205 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:07:47.205 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:07:47.206 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:47.208 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:07:47.208 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:07:47.208 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:07:47.208 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:07:47.208 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:07:47.208 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:07:47.209 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:07:47.209 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:07:47.209 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:07:47.209 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:07:47.209 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:07:47.209 INFO:tasks.workunit.client.0.smithi105.stderr:208496: start osd.5 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:07:48.480 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:07:48.481 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: _ 2024-09-17T19:07:48.482 INFO:tasks.workunit.client.0.smithi105.stderr:208498: hinfo_key 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: snapset 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:07:48.483 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:07:48.484 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:07:48.484 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:07:48.484 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:07:48.484 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:07:48.484 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:07:48.484 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:48.485 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:07:48.486 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:07:48.487 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:07:48.487 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:07:48.487 INFO:tasks.workunit.client.0.smithi105.stderr:208498: start osd.9 2024-09-17T19:07:48.487 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 very_ops 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2024-09-17T19:07:47.236+0000 7f2c999717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:53.027 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2024-09-17T19:07:47.240+0000 7f2c999717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2024-09-17T19:07:47.244+0000 7f2c999717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 0 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:53.028 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2024-09-17T19:07:49.004+0000 7f2c999717c0 -1 Falling back to public interface 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 1 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.029 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 3 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:53.030 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.988 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2024-09-17T19:07:53.024+0000 7f2c--osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:07:53.988 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:07:53.988 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:07:53.988 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:07:53.988 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:07:53.988 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2024-09-17T19:07:48.516+0000 7f10875bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2024-09-17T19:07:48.592+0000 7f10875bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2024-09-17T19:07:48.592+0000 7f10875bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 0 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.989 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2024-09-17T19:07:49.864+0000 7f10875bc7c0 -1 Falling back to public interface 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 1 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.990 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 3 2024-09-17T19:07:53.991 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:53.992 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:53.992 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2024-09-17T19:07:53.984+0000 7f10999717c0 -1 osd.5 100 log_to_monitors true 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 4 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:07:54.629 INFO:tasks.workunit.client.0.smithi105.stderr:208496: osd.5 up in weight 1 up_from 105 up_thru 96 down_at 101 last_clean_interval [96,100) [v2:127.0.0.1:6842/3916686083,v1:127.0.0.1:6843/3916686083] [v2:127.0.0.1:6844/3916686083,v1:127.0.0.1:6845/3916686083] exists,up ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:07:54.630 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 1 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 2 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 3 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 4 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 5 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 6 2024-09-17T19:07:54.631 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 7 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 8 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: 9' 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-17T19:07:54.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-17T19:07:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in 875bc7c0 -1 osd.9 100 log_to_monitors true 2024-09-17T19:07:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 4 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 5 2024-09-17T19:07:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: osd.9 up in weight 1 up_from 108 up_thru 0 down_at 101 last_clean_interval [98,100) [v2:127.0.0.1:6874/1160157025,v1:127.0.0.1:6875/1160157025] [v2:127.0.0.1:6876/1160157025,v1:127.0.0.1:6877/1160157025] exists,up 4bce93e9-bde2-482a-a639-925aec3e7f95 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:07:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:208498: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 1 2024-09-17T19:07:56.635 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 2 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 3 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 4 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 5 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 6 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 7 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 8 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: 9' 2024-09-17T19:07:56.636 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalon$ids 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673058 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673058 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058' 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:56.649 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509536 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509536 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536' 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645341 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645341 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341' 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182481 2024-09-17T19:07:56.650 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182481 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341 4-107374182481' 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341 4-107374182481 5-450971566082' 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855429 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855429 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341 4-107374182481 5-450971566082 6-150323855429' 2024-09-17T19:07:56.651 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:56.652 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:07:57.483 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691897 2024-09-17T19:07:57.483 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691897 2024-09-17T19:07:57.483 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341 4-107374182481 5-450971566082 6-150323855429 7-171798691897' 2024-09-17T19:07:57.483 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.483 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528367 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528367 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341 4-107374182481 5-450971566082 6-150323855429 7-171798691897 8-193273528367' 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467970 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467970 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673058 2-64424509536 3-309237645341 4-107374182481 5-450971566082 6-150323855429 7-171798691897 8-193273528367 9-463856467970' 2024-09-17T19:07:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.0 seq 21474836591 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:07:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836591 -lt 21474836591 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673058 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673058 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673058 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-17T19:07:57.486 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.1 seq 42949673058 2024-09-17T19:07:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22e/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:07:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836592 2024-09-17T19:07:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836592 2024-09-17T19:07:57.909 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592' 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673060 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673060 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060' 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509537 2024-09-17T19:07:57.910 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509537 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537' 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=309237645342 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 309237645342 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342' 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182482 2024-09-17T19:07:57.911 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182482 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342 4-107374182482' 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566083 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566083 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342 4-107374182482 5-450971566083' 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:57.912 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:07:59.022 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855431 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855431 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342 4-107374182482 5-450971566083 6-150323855431' 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691898 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691898 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342 4-107374182482 5-450971566083 6-150323855431 7-171798691898' 2024-09-17T19:07:59.023 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528369 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528369 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342 4-107374182482 5-450971566083 6-150323855431 7-171798691898 8-193273528369' 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:07:59.024 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467971 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467971 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673060 2-64424509537 3-309237645342 4-107374182482 5-450971566083 6-150323855431 7-171798691898 8-193273528369 9-463856467971' 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836592 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836592 2024-09-17T19:07:59.025 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836592 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.0 seq 21474836592 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836592 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673060 2024-09-17T19:07:59.026 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/stan22: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509536 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509536 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509536 2024-09-17T19:07:59.213 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.2 seq 64424509536 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509536 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645341 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645341 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:59.214 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645341 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645341' 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.3 seq 309237645341 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645342 -lt 309237645341 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182481 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182481 2024-09-17T19:07:59.215 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182481 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.4 seq 107374182481 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182482 -lt 107374182481 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:07:59.216 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-450971566082 2024-09-17T19:08:00.100 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //dalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:08:00.100 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673060 2024-09-17T19:08:00.100 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673060 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.1 seq 42949673060 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673060 -lt 42949673060 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509537 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509537 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509537 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.2 seq 64424509537 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509537 -lt 64424509537 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-309237645342 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-309237645342 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=309237645342 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 309237645342' 2024-09-17T19:08:00.102 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.3 seq 309237645342 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 309237645343 -lt 309237645342 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182482 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182482 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.103 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182482 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephthome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-450971566082 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 450971566082' 2024-09-17T19:08:00.239 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.5 seq 450971566082 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566082 -lt 450971566082 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855429 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855429 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855429 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-09-17T19:08:00.240 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.6 seq 150323855429 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855429 -lt 150323855429 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691897 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691897 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691897 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.7 seq 171798691897 2024-09-17T19:08:00.241 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:08:00.242 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691898 -lt 171798691897 2024-09-17T19:08:00.242 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:00.242 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528367 2024-09-17T19:08:00.242 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:00.242 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:08:00.242 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528367 2024-09-17T19:08:01.439 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flusest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-09-17T19:08:01.439 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.4 seq 107374182482 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182483 -lt 107374182482 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-450971566083 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:08:01.440 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-450971566083 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566083 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 450971566083' 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.5 seq 450971566083 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566084 -lt 450971566083 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855431 2024-09-17T19:08:01.441 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855431 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855431 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.6 seq 150323855431 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855431 -lt 150323855431 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:01.442 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691898 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691898 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691898 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.7 seq 171798691898 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:08:01.443 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691898 -lt 171798691898 2024-09-17T19:08:02.083 INFO:tasks.workunit.client.0.smithi105.stderr:208498h_pg_stats: cut -d - -f 2 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528367 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.8 seq 193273528367 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528369 -lt 193273528367 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467970 2024-09-17T19:08:02.084 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467970 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467970 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467970' 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: waiting osd.9 seq 463856467970 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467971 -lt 463856467970 2024-09-17T19:08:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:208496: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:08:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 208495 2024-09-17T19:08:02.918 INFO:tasks.workunit.client.0.smithi105.stderr:: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:02.918 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528369 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528369 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528369 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.8 seq 193273528369 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:08:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528369 -lt 193273528369 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467971 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467971 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467971 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467971' 2024-09-17T19:08:02.920 INFO:tasks.workunit.client.0.smithi105.stderr:208498: waiting osd.9 seq 463856467971 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467972 -lt 463856467971 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:02.921 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:02.922 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:02.922 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:02.922 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:02.922 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:02.922 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:08:03.293 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:03.293 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:03.293 INFO:tasks.workunit.client.0.smithi105.stderr:208498: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:208498: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:08:03.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:08:03.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:08:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-17T19:08:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T19:08:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T19:08:03.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 211453"' 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 211453' 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:08:03.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 211455"' 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 211455' 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 211453 211455' 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 211453 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/211456: /' 2024-09-17T19:08:03.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:08:03.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/211458: /' 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:08:07.436 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:08:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:08:07.438 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:08:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:08:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: start osd.3 2024-09-17T19:08:07.442 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:2211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:08:07.838 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:08:07.839 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:08:07.840 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:07.842 INFO:tasks.workunit.client.0.smithi105.stderr:211458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:08:07.843 INFO:tasks.workunit.client.0.smithi105.stderr:211458: start osd.5 2024-09-17T19:08:07.844 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:08:12.485 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T19:08:12.485 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T19:08:12.485 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:08:12.485 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2024-09-17T19:08:07.471+0000 7f97aaa137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2024-09-17T19:08:07.475+0000 7f97aaa137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2024-09-17T19:08:07.475+0000 7f97aaa137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 0 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.486 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2024-09-17T19:08:09.003+0000 7f97aaa137c0 -1 Falling back to public interface 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 1 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2 2024-09-17T19:08:12.487 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 3 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:12.488 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.613 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2024-09-17T19:08:12.483+0000 7f97aaa137c0 -11458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:08:12.613 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:08:12.613 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:08:12.613 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:08:12.613 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:08:12.613 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2024-09-17T19:08:07.867+0000 7f6d0e5f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2024-09-17T19:08:07.919+0000 7f6d0e5f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2024-09-17T19:08:07.919+0000 7f6d0e5f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 0 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:12.614 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2024-09-17T19:08:08.979+0000 7f6d0e5f77c0 -1 Falling back to public interface 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 1 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2 2024-09-17T19:08:12.615 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2024-09-17T19:08:11.915+0000 7f6d0e5f77c0 -1 osd.5 109 log_to_monitors true 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 3 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:12.616 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:13.629 INFO:tasks.workunit.client.0.smithi105.stderr:211458: osd.5 up in weight 1 up_from 113 up_thru 105 down_at 110 last_clean_interval [105,109) [v2:127.0.0.1:6826/3253525070,v1:127.0.0.1:6827/3253525070] [v2:127.0.0.1:6828/3253525070,v1:127.0.0.1:6829/3253525070] exists,up ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:08:13.629 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:08:13.629 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:08:13.629 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:08:13.630 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 1 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 2 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 3 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 4 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 5 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 6 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 7 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 8 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: 9' 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:08:13.631 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836596 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836596 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596' 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673064 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673064 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064' 2024-09-17T19:08:13.632 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.clien1 osd.3 109 log_to_monitors true 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 4 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: osd.3 up in weight 1 up_from 115 up_thru 108 down_at 110 last_clean_interval [72,109) [v2:127.0.0.1:6842/3768319382,v1:127.0.0.1:6843/3768319382] [v2:127.0.0.1:6844/3768319382,v1:127.0.0.1:6845/3768319382] exists,up 4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:08:14.644 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:08:14.645 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 1 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 2 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 3 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 4 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 5 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 6 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 7 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 8 2024-09-17T19:08:14.646 INFO:tasks.workunit.client.0.smithi105.stderr:211456: 9' 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836598 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836598 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598' 2024-09-17T19:08:14.647 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:2114t.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509542 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509542 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542' 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239042 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239042 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042' 2024-09-17T19:08:15.159 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182487 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182487 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042 4-107374182487' 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304450 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304450 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042 4-107374182487 5-485331304450' 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855435 2024-09-17T19:08:15.160 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855435 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042 4-107374182487 5-485331304450 6-150323855435' 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691903 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691903 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042 4-107374182487 5-485331304450 6-150323855435 7-171798691903' 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.161 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:08:15.747 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest56: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:08:15.747 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673065 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673065 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065' 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509543 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509543 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543' 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=493921239043 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 493921239043 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043' 2024-09-17T19:08:15.748 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182488 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182488 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043 4-107374182488' 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=485331304451 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 485331304451 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043 4-107374182488 5-485331304451' 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855436 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855436 2024-09-17T19:08:15.749 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043 4-107374182488 5-485331304451 6-150323855436' 2024-09-17T19:08:15.750 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:15.750 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:08:16.044 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: f/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528373 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528373 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042 4-107374182487 5-485331304450 6-150323855435 7-171798691903 8-193273528373' 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467976 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467976 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836596 1-42949673064 2-64424509542 3-493921239042 4-107374182487 5-485331304450 6-150323855435 7-171798691903 8-193273528373 9-463856467976' 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836596 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:08:16.045 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836596 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836596 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.0 seq 21474836596 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836596 -lt 21474836596 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673064 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673064 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673064 2024-09-17T19:08:16.046 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.1 seq 42949673064 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673065 -lt 42949673064 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509542 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:16.047 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:08:16.424 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clush_pg_stats: seq=171798691904 2024-09-17T19:08:16.424 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691904 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043 4-107374182488 5-485331304451 6-150323855436 7-171798691904' 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528374 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528374 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043 4-107374182488 5-485331304451 6-150323855436 7-171798691904 8-193273528374' 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467977 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467977 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836598 1-42949673065 2-64424509543 3-493921239043 4-107374182488 5-485331304451 6-150323855436 7-171798691904 8-193273528374 9-463856467977' 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:16.425 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836598 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836598 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836598 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836598' 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.0 seq 21474836598 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836598 -lt 21474836598 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673065 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:16.426 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:08:16.427 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673065 2024-09-17T19:08:16.427 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:16.427 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673065 2024-09-17T19:08:16.427 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-09-17T19:08:16.427 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.1 seq 42949673065 2024-09-17T19:08:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flushlone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509542 2024-09-17T19:08:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509542 2024-09-17T19:08:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509542' 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.2 seq 64424509542 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509543 -lt 64424509542 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-493921239042 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-493921239042 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239042 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 493921239042' 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.3 seq 493921239042 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:08:17.119 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239042 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182487 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182487 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182487 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.4 seq 107374182487 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182488 -lt 107374182487 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.120 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-485331304450 2024-09-17T19:08:17.121 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:17.121 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:08:17.121 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-485331304450 2024-09-17T19:08:17.121 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.121 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304450 2024-09-17T19:08:17.730 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo _pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:08:17.730 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673065 -lt 42949673065 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509543 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509543 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509543 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.2 seq 64424509543 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509543 -lt 64424509543 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.731 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-493921239043 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-493921239043 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=493921239043 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 493921239043' 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.3 seq 493921239043 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 493921239043 -lt 493921239043 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182488 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:08:17.732 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182488 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182488 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.4 seq 107374182488 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182488 -lt 107374182488 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:17.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cep'waiting osd.5 seq 485331304450' 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.5 seq 485331304450 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304451 -lt 485331304450 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855435 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.465 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855435 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855435 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855435' 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.6 seq 150323855435 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855436 -lt 150323855435 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691903 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691903 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:18.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691903 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691903' 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.7 seq 171798691903 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691904 -lt 171798691903 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528373 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528373 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528373 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528373' 2024-09-17T19:08:18.467 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.8 seq 193273528373 2024-09-17T19:08:18.468 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:08:18.468 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528375 -lt 193273528373 2024-09-17T19:08:18.733 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-485331304451 2024-09-17T19:08:18.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:08:18.733 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-485331304451 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=485331304451 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 485331304451' 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.5 seq 485331304451 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 485331304452 -lt 485331304451 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855436 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855436 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855436 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855436' 2024-09-17T19:08:18.734 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.6 seq 150323855436 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855436 -lt 150323855436 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691904 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691904 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691904 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.7 seq 171798691904 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:08:18.735 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691904 -lt 171798691904 2024-09-17T19:08:18.736 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:18.736 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:18.736 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528374 2024-09-17T19:08:18.736 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:08:18.736 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528374 2024-09-17T19:08:20.057 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_statush_pg_stats: for s in $seqs 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467976 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467976 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467976 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467976' 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: waiting osd.9 seq 463856467976 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:08:20.058 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467977 -lt 463856467976 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:20.059 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 5 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:20.060 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //hos: cut -d - -f 2 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528374 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.8 seq 193273528374 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528375 -lt 193273528374 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467977 2024-09-17T19:08:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467977 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467977 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467977' 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: waiting osd.9 seq 463856467977 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467977 -lt 463856467977 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:20.467 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:20.468 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:08:20.469 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:08:20.469 INFO:tasks.workunit.client.0.smithi105.stderr:211456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:08:20.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:08:20.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:08:20.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 211455 2024-09-17T19:08:20.747 INFO:tasks.workunit.client.0.smithi105.stderr:me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:20.747 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:20.747 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:20.747 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:20.747 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:211458: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:08:20.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T19:08:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:08:20.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T19:08:20.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:08:20.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:08:20.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:08:20.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:08:21.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T19:08:21.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T19:08:21.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:08:21.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:08:21.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:08:21.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:08:21.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:08:21.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:08:21.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:07:39.225512+0000 2024-09-17T19:08:21.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:08:21.700 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:08:21.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:07:39.225512+0000 2024-09-17T19:08:21.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:08:21.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:07:39.225512+0000 2024-09-17T19:08:21.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:08:21.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:08:21.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:08:21.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:08:21.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:08:21.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:08:21.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:08:21.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:08:22.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:07:39.225512+0000 '>' 2024-09-17T19:07:39.225512+0000 2024-09-17T19:08:22.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:08:23.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:08:23.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:08:23.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:08:23.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:08:23.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:08:23.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:08:23.027 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:08:23.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:07:39.225512+0000 '>' 2024-09-17T19:07:39.225512+0000 2024-09-17T19:08:23.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:08:24.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:08:24.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:08:24.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:08:24.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:08:24.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:08:24.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:08:24.371 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:08:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:08:22.521265+0000 '>' 2024-09-17T19:07:39.225512+0000 2024-09-17T19:08:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:08:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:08:26.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T19:08:26.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:08:26.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:08:26.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 214905"' 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 214905' 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 214907"' 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 214907' 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/214908: /' 2024-09-17T19:08:26.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 214905 214907' 2024-09-17T19:08:26.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:08:26.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:08:26.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 214905 2024-09-17T19:08:26.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:08:26.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/214910: /' 2024-09-17T19:08:27.931 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:08:27.931 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:08:27.932 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: _ 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: hinfo_key 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: snapset 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:08:27.933 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:08:27.934 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:08:27.934 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:08:27.934 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:08:27.934 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:08:27.934 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:08:27.936 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:27.937 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:08:27.938 INFO:tasks.workunit.client.0.smithi105.stderr:214910: start osd.5 2024-09-17T19:08:28.013 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:08:28.013 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:08:28.014 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: _ 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: hinfo_key 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: snapset 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:08:28.015 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:08:28.016 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:08:28.017 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T19:08:28.018 INFO:tasks.workunit.client.0.smithi105.stderr:214908: start osd.3 2024-09-17T19:08:32.969 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2024-09-17T19:08:28.039+0000 7f505cd5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2024-09-17T19:08:28.043+0000 7f505cd5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2024-09-17T19:08:28.043+0000 7f505cd5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:08:32.970 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 0 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2024-09-17T19:08:29.375+0000 7f505cd5e7c0 -1 Falling back to public interface 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 1 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:32.971 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 3 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:33.583 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-17T19:08:33.583 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:08:33.583 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2024-09-17T19:08:27.967+0000 7f758b4677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2024-09-17T19:08:27.987+0000 7f758b4677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2024-09-17T19:08:27.987+0000 7f758b4677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:08:33.584 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 0 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2024-09-17T19:08:29.519+0000 7f758b4677c0 -1 Falling back to public interface 2024-09-17T19:08:33.585 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 1 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:33.586 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 3 2024-09-17T19:08:33.587 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:33.588 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:33.588 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:35.200 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2024-09-17T19:08:33.579+0000 7f750/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:35.200 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2024-09-17T19:08:33.607+0000 7f505cd5e7c0 -1 osd.3 116 log_to_monitors true 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 4 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: osd.3 up in weight 1 up_from 121 up_thru 115 down_at 117 last_clean_interval [115,116) [v2:127.0.0.1:6826/2725265020,v1:127.0.0.1:6827/2725265020] [v2:127.0.0.1:6828/2725265020,v1:127.0.0.1:6829/2725265020] exists,up 4846fc06-1303-474b-b310-d67da1bf6660 2024-09-17T19:08:35.201 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:08:35.202 INFO:tasks.workunit.client.0.smithi105.stderr:214908: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:08:35.203 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 1 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 2 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 3 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 4 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 5 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 6 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 7 2024-09-17T19:08:35.204 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 8 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: 9' 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836603 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836603 2024-09-17T19:08:35.205 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603' 2024-09-17T19:08:36.353 INFO:tasks.workunit.client.0.smithi105.stderr:2149088b4677c0 -1 osd.5 116 log_to_monitors true 2024-09-17T19:08:36.353 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:36.353 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:36.353 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:08:36.353 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 4 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 5 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: osd.5 up in weight 1 up_from 121 up_thru 121 down_at 117 last_clean_interval [113,116) [v2:127.0.0.1:6842/1478829950,v1:127.0.0.1:6843/1478829950] [v2:127.0.0.1:6844/1478829950,v1:127.0.0.1:6845/1478829950] exists,up ef4552e6-877d-496f-90d6-eb71ad7992f3 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:08:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:08:36.355 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 1 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 2 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 3 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 4 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 5 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 6 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 7 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 8 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: 9' 2024-09-17T19:08:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:08:36.504 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standa: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673070 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673070 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070' 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509548 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509548 2024-09-17T19:08:36.505 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548' 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042818 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042818 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818' 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182493 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182493 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818 4-107374182493' 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:08:36.506 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042818 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042818 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818 4-107374182493 5-519691042818' 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855441 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855441 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818 4-107374182493 5-519691042818 6-150323855441' 2024-09-17T19:08:36.507 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:08:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691909 2024-09-17T19:08:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691909 2024-09-17T19:08:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818 4-107374182493 5-519691042818 6-150323855441 7-171798691909' 2024-09-17T19:08:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.229 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528379 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528379 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818 4-107374182493 5-519691042818 6-150323855441 7-171798691909 8-193273528379' 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467982 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467982 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836603 1-42949673070 2-64424509548 3-519691042818 4-107374182493 5-519691042818 6-150323855441 7-171798691909 8-193273528379 9-463856467982' 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836603 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:08:37.230 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836603 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836603 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.0 seq 21474836603 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836603 -lt 21474836603 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673070 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673070 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673070 2024-09-17T19:08:37.231 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673070' 2024-09-17T19:08:37.318 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waitilone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836604 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836604 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604' 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673071 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673071 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071' 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:08:37.319 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509549 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509549 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549' 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042819 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042819 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819' 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182494 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182494 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819 4-107374182494' 2024-09-17T19:08:37.320 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:08:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042819 2024-09-17T19:08:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042819 2024-09-17T19:08:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819 4-107374182494 5-519691042819' 2024-09-17T19:08:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:37.321 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:08:38.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855442 2024-09-17T19:08:38.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855442 2024-09-17T19:08:38.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819 4-107374182494 5-519691042819 6-150323855442' 2024-09-17T19:08:38.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:38.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691910 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691910 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819 4-107374182494 5-519691042819 6-150323855442 7-171798691910' 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528381 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528381 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819 4-107374182494 5-519691042819 6-150323855442 7-171798691910 8-193273528381' 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:08:38.120 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=463856467983 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 463856467983 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836604 1-42949673071 2-64424509549 3-519691042819 4-107374182494 5-519691042819 6-150323855442 7-171798691910 8-193273528381 9-463856467983' 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836604 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836604 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:38.121 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836604 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.0 seq 21474836604 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836604 -lt 21474836604 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673071 2024-09-17T19:08:38.122 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:38.620 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/sng osd.1 seq 42949673070 2024-09-17T19:08:38.620 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:08:38.620 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673070 -lt 42949673070 2024-09-17T19:08:38.620 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:38.620 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509548 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509548 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509548 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.2 seq 64424509548 2024-09-17T19:08:38.621 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509549 -lt 64424509548 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-519691042818 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-519691042818 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:38.622 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042818 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 519691042818' 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.3 seq 519691042818 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042819 -lt 519691042818 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182493 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:38.623 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182493 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182493 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182493' 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.4 seq 107374182493 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182494 -lt 107374182493 2024-09-17T19:08:38.624 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.115 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtandalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:08:39.115 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673071 2024-09-17T19:08:39.115 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.115 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673071 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.1 seq 42949673071 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673072 -lt 42949673071 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509549 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:08:39.116 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509549 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509549 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.2 seq 64424509549 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509549 -lt 64424509549 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-519691042819 2024-09-17T19:08:39.117 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-519691042819 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042819 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 519691042819' 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.3 seq 519691042819 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042819 -lt 519691042819 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.118 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182494 2024-09-17T19:08:39.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:08:39.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182494 2024-09-17T19:08:39.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.119 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182494 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cetest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-519691042818 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-519691042818 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042818 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 519691042818' 2024-09-17T19:08:39.576 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.5 seq 519691042818 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042819 -lt 519691042818 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855441 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855441 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.577 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855441 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855441' 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.6 seq 150323855441 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855442 -lt 150323855441 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691909 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691909 2024-09-17T19:08:39.578 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691909 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691909' 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.7 seq 171798691909 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691910 -lt 171798691909 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528379 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:39.579 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:08:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-09-17T19:08:40.409 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.4 seq 107374182494 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182494 -lt 107374182494 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-519691042819 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-519691042819 2024-09-17T19:08:40.410 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042819 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 519691042819' 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.5 seq 519691042819 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042819 -lt 519691042819 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855442 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:08:40.411 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855442 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855442 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855442' 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.6 seq 150323855442 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855443 -lt 150323855442 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691910 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:40.412 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691910 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691910 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.7 seq 171798691910 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:08:40.413 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691910 -lt 171798691910 2024-09-17T19:08:41.326 INFO:tasks.workunit.client.0.smithi105.stderr:214o 8-193273528379 2024-09-17T19:08:41.326 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:41.326 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528379 2024-09-17T19:08:41.326 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-09-17T19:08:41.326 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.8 seq 193273528379 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528381 -lt 193273528379 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467982 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467982 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467982 2024-09-17T19:08:41.327 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467982' 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: waiting osd.9 seq 463856467982 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467983 -lt 463856467982 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:41.328 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:08:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:08:41.330 INFO:tasks.workunit.client.0.smithi105.stderr:214908: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:08:41.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:08:41.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:08:41.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 214907 2024-09-17T19:08:41.804 INFO:tasks.workunit.client.0.smithi105.stderr:910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:41.804 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528381 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528381 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528381 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.8 seq 193273528381 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528381 -lt 193273528381 2024-09-17T19:08:41.805 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-463856467983 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-463856467983 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=463856467983 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 463856467983' 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: waiting osd.9 seq 463856467983 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:08:41.806 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 463856467983 -lt 463856467983 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:08:41.807 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:08:41.808 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:08:41.808 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:08:42.175 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:08:42.175 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:08:42.175 INFO:tasks.workunit.client.0.smithi105.stderr:214910: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:08:42.175 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:08:42.175 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:08:42.175 INFO:tasks.workunit.client.0.smithi105.stderr:214910: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:08:42.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:08:42.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:08:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T19:08:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:08:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:08:42.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:08:42.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:08:42.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:08:42.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:08:42.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:08:42.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:08:42.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:08:42.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:08:42.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:08:42.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:08:42.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:08:42.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:08:42.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:08:42.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:08:42.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:08:42.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:08:42.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:08:42.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:08:42.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:08:42.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:08:42.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:08:42.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:42.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:42.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:08:42.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:08:42.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:08:42.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T19:08:42.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:08:42.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:08:42.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:08:42.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:08:42.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:08:42.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:08:42.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:08:42.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:08:42.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:08:42.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:08:42.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:08:42.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:08:42.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:08:42.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:08:42.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:08:42.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:08:42.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:08:42.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:42.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:42.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:08:42.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:08:42.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:08:42.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T19:08:42.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T19:08:42.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:42.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:42.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T19:08:42.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T19:08:42.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-09-17T19:08:42.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-09-17T19:08:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T19:08:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T19:08:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T19:08:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T19:08:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T19:08:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T19:08:43.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T19:08:43.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:43.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:43.110 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:43.110 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:43.110 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:43.111 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:43.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T19:08:43.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T19:08:43.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T19:08:43.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:08:43.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:08:43.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T19:08:43.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T19:08:43.316 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:08:43.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:08:43.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:08:43.317 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:08:43.317 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:43.317 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:43.317 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:08:43.318 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:08:43.318 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T19:08:43.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T19:08:43.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:08:43.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:08:43.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T19:08:43.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T19:08:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:08:43.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T19:08:43.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-09-17T19:08:43.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T19:08:43.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T19:08:43.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T19:08:43.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T19:08:43.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T19:08:43.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T19:08:44.387 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T19:08:44.387 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:44.387 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:44.388 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:44.388 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:44.388 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:44.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:44.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:08:44.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:08:44.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:08:44.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:08:44.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:08:44.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:08:44.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:08:44.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:08:44.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:08:44.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:08:44.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:08:44.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:08:44.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cfe32843-12e3-481e-8aad-3eb0e93e3933 2024-09-17T19:08:44.432 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 cfe32843-12e3-481e-8aad-3eb0e93e3933 2024-09-17T19:08:44.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 cfe32843-12e3-481e-8aad-3eb0e93e3933' 2024-09-17T19:08:44.433 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:08:44.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC80+lmTQqjGhAAeOalvvZMONrtwCEWMQB2Ow== 2024-09-17T19:08:44.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC80+lmTQqjGhAAeOalvvZMONrtwCEWMQB2Ow=="}' 2024-09-17T19:08:44.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cfe32843-12e3-481e-8aad-3eb0e93e3933 -i td/osd-scrub-repair/0/new.json 2024-09-17T19:08:44.698 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:08:44.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T19:08:44.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC80+lmTQqjGhAAeOalvvZMONrtwCEWMQB2Ow== --osd-uuid cfe32843-12e3-481e-8aad-3eb0e93e3933 2024-09-17T19:08:44.747 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:44.743+0000 7fc104bfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:44.750 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:44.747+0000 7fc104bfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:44.751 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:44.747+0000 7fc104bfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:44.751 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:44.747+0000 7fc104bfd7c0 -1 bdev(0x55699229bc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T19:08:44.751 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:44.747+0000 7fc104bfd7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T19:08:51.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T19:08:51.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:08:51.026 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T19:08:51.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T19:08:51.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:08:51.422 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T19:08:51.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T19:08:51.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:08:51.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:08:51.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:08:51.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:08:51.454 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:51.451+0000 7fafcf81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:51.456 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:51.451+0000 7fafcf81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:51.457 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:51.455+0000 7fafcf81b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:51.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:08:51.745 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:08:51.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:51.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:08:52.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:52.776 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:52.771+0000 7fafcf81b7c0 -1 Falling back to public interface 2024-09-17T19:08:53.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:53.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:53.080 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:08:53.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:08:53.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:53.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:08:53.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:54.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:54.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:54.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:08:54.420 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:08:54.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:54.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:08:54.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:55.587 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:08:55.582+0000 7fafcf81b7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T19:08:55.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:55.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:55.756 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:08:55.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:08:55.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:55.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:08:56.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:57.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:57.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:57.112 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:08:57.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:08:57.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:57.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:08:57.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:58.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:58.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:58.446 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:08:58.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:08:58.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:58.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:08:58.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:08:59.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:08:59.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:08:59.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:08:59.768 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:08:59.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:08:59.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:00.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:01.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:01.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:01.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:09:01.103 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:09:01.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:01.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:01.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:02.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:02.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:02.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:09:02.458 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:09:02.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:02.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:02.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:03.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:03.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:03.785 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:09:03.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:09:03.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:03.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:04.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:05.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:05.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:05.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:09:05.121 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:09:05.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:05.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:05.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:06.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:06.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:06.457 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:09:06.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:09:06.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:06.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:06.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:07.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:07.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:07.779 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:09:07.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:09:07.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:07.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:08.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:09.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:09.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:09.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:09:09.103 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:09:09.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:09.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:09.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:10.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:10.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:10.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:09:10.670 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:09:10.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:10.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:11.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:12.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:12.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:12.091 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:09:12.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:09:12.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:12.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:12.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:13.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:13.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:13.679 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:09:13.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:09:13.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:13.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:14.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:15.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:15.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:15.378 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:09:15.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:09:15.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:15.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:15.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:16.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:16.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:16.954 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:09:16.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:09:16.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:16.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:17.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:18.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:18.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:18.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:09:18.361 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:09:18.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:18.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:18.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:19.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:19.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:19.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:09:19.740 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:09:19.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:19.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:09:20.089 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1435885375,v1:127.0.0.1:6803/1435885375] [v2:127.0.0.1:6804/1435885375,v1:127.0.0.1:6805/1435885375] exists,up cfe32843-12e3-481e-8aad-3eb0e93e3933 2024-09-17T19:09:20.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:09:20.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:09:20.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:09:20.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:09:20.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:09:20.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:09:20.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:09:20.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:09:20.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0d40ede1-5563-4cf3-825a-e420f43eff32 2024-09-17T19:09:20.098 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 0d40ede1-5563-4cf3-825a-e420f43eff32 2024-09-17T19:09:20.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0d40ede1-5563-4cf3-825a-e420f43eff32' 2024-09-17T19:09:20.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:09:20.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDg0+lmz8TiBhAAhTLDGCAMEclvBVhVxlN3Ug== 2024-09-17T19:09:20.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDg0+lmz8TiBhAAhTLDGCAMEclvBVhVxlN3Ug=="}' 2024-09-17T19:09:20.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0d40ede1-5563-4cf3-825a-e420f43eff32 -i td/osd-scrub-repair/1/new.json 2024-09-17T19:09:20.480 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:09:20.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T19:09:20.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDg0+lmz8TiBhAAhTLDGCAMEclvBVhVxlN3Ug== --osd-uuid 0d40ede1-5563-4cf3-825a-e420f43eff32 2024-09-17T19:09:20.533 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:20.530+0000 7f9a032317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:20.574 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:20.570+0000 7f9a032317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:20.575 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:20.570+0000 7f9a032317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:20.576 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:20.570+0000 7f9a032317c0 -1 bdev(0x55da823b3c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T19:09:20.576 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:20.570+0000 7f9a032317c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T19:09:26.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T19:09:26.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:09:26.937 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T19:09:26.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T19:09:26.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:09:27.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T19:09:27.345 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T19:09:27.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:09:27.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:09:27.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:09:27.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:09:27.377 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:27.374+0000 7f25d954a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:27.403 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:27.398+0000 7f25d954a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:27.405 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:27.402+0000 7f25d954a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:27.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T19:09:27.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:09:27.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:09:27.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:09:27.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:09:27.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:27.698 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:09:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:09:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:27.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:28.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:28.688 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:28.682+0000 7f25d954a7c0 -1 Falling back to public interface 2024-09-17T19:09:29.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:29.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:29.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:09:29.039 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:09:29.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:29.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:29.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:30.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:30.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:30.372 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:09:30.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:09:30.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:30.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:30.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:31.486 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:31.482+0000 7f25d954a7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T19:09:31.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:31.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:31.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:09:31.719 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:09:31.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:31.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:32.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:33.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:33.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:33.101 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:09:33.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:09:33.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:33.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:33.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:34.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:34.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:34.540 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:09:34.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:09:34.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:34.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:34.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:35.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:35.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:35.879 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:09:35.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:09:35.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:35.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:36.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:37.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:37.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:37.208 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:09:37.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:09:37.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:37.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:37.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:38.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:38.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:38.562 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:09:38.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:09:38.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:38.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:38.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:39.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:39.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:39.887 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:09:39.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:09:39.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:39.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:40.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:41.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:41.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:41.251 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:09:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:09:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:41.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:42.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:42.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:42.709 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:09:42.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:09:42.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:42.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:43.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:44.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:44.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:44.033 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:09:44.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:09:44.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:44.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:44.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:45.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:45.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:45.789 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:09:45.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:09:45.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:45.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:46.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:47.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:47.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:47.578 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:09:47.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:09:47.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:47.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:47.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:48.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:48.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:48.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:09:48.979 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:09:48.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:48.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:49.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:50.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:50.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:50.427 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:09:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:09:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:50.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:50.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:51.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:51.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:51.937 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:09:51.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:09:51.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:51.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:52.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:53.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:53.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:53.703 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:09:53.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:09:53.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:53.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:54.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:09:55.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:09:55.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:09:55.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:09:55.096 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:09:55.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:09:55.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:09:55.438 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2626201808,v1:127.0.0.1:6811/2626201808] [v2:127.0.0.1:6812/2626201808,v1:127.0.0.1:6813/2626201808] exists,up 0d40ede1-5563-4cf3-825a-e420f43eff32 2024-09-17T19:09:55.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:09:55.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:09:55.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:09:55.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:09:55.444 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:09:55.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=23dd438e-f76a-46c5-ba9b-d113891ac76d 2024-09-17T19:09:55.445 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 23dd438e-f76a-46c5-ba9b-d113891ac76d 2024-09-17T19:09:55.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 23dd438e-f76a-46c5-ba9b-d113891ac76d' 2024-09-17T19:09:55.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:09:55.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAD1Olm0GTMGxAAmUYPTdpw3mkwYmdqumjJ3Q== 2024-09-17T19:09:55.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAD1Olm0GTMGxAAmUYPTdpw3mkwYmdqumjJ3Q=="}' 2024-09-17T19:09:55.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 23dd438e-f76a-46c5-ba9b-d113891ac76d -i td/osd-scrub-repair/2/new.json 2024-09-17T19:09:55.949 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:09:55.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T19:09:55.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAD1Olm0GTMGxAAmUYPTdpw3mkwYmdqumjJ3Q== --osd-uuid 23dd438e-f76a-46c5-ba9b-d113891ac76d 2024-09-17T19:09:56.008 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:56.005+0000 7f1c025d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:56.050 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:56.049+0000 7f1c025d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:56.052 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:56.049+0000 7f1c025d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:09:56.053 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:56.053+0000 7f1c025d67c0 -1 bdev(0x55bdb011bc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T19:09:56.053 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:09:56.053+0000 7f1c025d67c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T19:10:03.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T19:10:03.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:10:03.032 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T19:10:03.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T19:10:03.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:10:03.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T19:10:03.429 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T19:10:03.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:10:03.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:10:03.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:10:03.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:10:03.461 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:03.461+0000 7f7099bd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:03.463 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:03.461+0000 7f7099bd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:03.464 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:03.465+0000 7f7099bd97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:03.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T19:10:03.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:10:03.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:03.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:04.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:04.779 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:04.777+0000 7f7099bd97c0 -1 Falling back to public interface 2024-09-17T19:10:05.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:05.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:05.128 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:10:05.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:10:05.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:05.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:05.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:06.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:06.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:06.447 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:10:06.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:10:06.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:06.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:06.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:07.461 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:07.461+0000 7f7099bd97c0 -1 osd.2 0 log_to_monitors true 2024-09-17T19:10:07.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:07.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:07.784 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:10:07.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:10:07.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:07.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:08.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:09.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:09.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:09.226 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:10:09.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:10:09.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:09.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:09.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:10.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:10.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:10.609 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:10:10.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:10:10.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:10.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:10.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:11.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:11.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:11.942 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:10:11.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:10:11.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:11.943 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:12.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:13.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:13.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:13.321 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:10:13.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:10:13.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:13.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:13.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:14.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:14.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:14.642 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:10:14.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:10:14.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:14.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:14.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:15.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:15.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:15.977 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:10:15.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:10:15.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:15.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:16.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:17.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:17.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:17.351 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:10:17.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:10:17.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:17.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:17.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:18.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:18.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:18.809 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:10:18.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:10:18.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:18.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:19.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:20.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:20.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:20.157 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:10:20.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:10:20.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:20.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:20.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:21.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:21.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:21.498 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:10:21.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:10:21.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:21.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:22.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:23.400 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:23.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:23.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:10:23.401 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:10:23.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:23.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:23.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:24.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:24.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:24.968 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:10:24.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:10:24.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:24.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:25.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:26.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:26.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:26.585 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:10:26.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:10:26.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:26.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:27.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:28.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:28.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:28.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:10:28.421 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:10:28.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:28.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:29.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:30.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:30.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:30.070 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:10:30.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:10:30.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:30.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:30.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:31.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:31.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:31.642 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:10:31.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:10:31.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:31.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:10:31.993 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2005500500,v1:127.0.0.1:6819/2005500500] [v2:127.0.0.1:6820/2005500500,v1:127.0.0.1:6821/2005500500] exists,up 23dd438e-f76a-46c5-ba9b-d113891ac76d 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:10:31.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:10:31.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:10:31.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:10:31.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:10:31.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:10:31.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:10:31.999 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:10:31.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:10:31.999 INFO:tasks.workunit.client.0.smithi105.stdout:add osd3 f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:10:32.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 f5ff2c0d-ce54-4109-a18b-7f2affb0093a' 2024-09-17T19:10:32.000 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:10:32.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAo1Olmuu0ZARAA945kvwIlpk2hM3Yp1r5EGQ== 2024-09-17T19:10:32.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAo1Olmuu0ZARAA945kvwIlpk2hM3Yp1r5EGQ=="}' 2024-09-17T19:10:32.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f5ff2c0d-ce54-4109-a18b-7f2affb0093a -i td/osd-scrub-repair/3/new.json 2024-09-17T19:10:32.372 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:10:32.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2024-09-17T19:10:32.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAo1Olmuu0ZARAA945kvwIlpk2hM3Yp1r5EGQ== --osd-uuid f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:10:32.425 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:32.425+0000 7f2fe25337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:32.457 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:32.457+0000 7f2fe25337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:32.457+0000 7f2fe25337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:32.457+0000 7f2fe25337c0 -1 bdev(0x5612d84d8800 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2024-09-17T19:10:32.459 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:32.457+0000 7f2fe25337c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-09-17T19:10:38.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-09-17T19:10:38.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:10:38.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-17T19:10:38.913 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd3 key to auth repository 2024-09-17T19:10:38.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:10:39.327 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.3 2024-09-17T19:10:39.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-17T19:10:39.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:10:39.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:10:39.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:10:39.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:10:39.360 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:39.357+0000 7f132c15a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:39.396 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:39.393+0000 7f132c15a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:39.397 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:39.397+0000 7f132c15a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:10:39.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:10:39.672 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:10:39.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:39.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:39.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:40.463 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:40.461+0000 7f132c15a7c0 -1 Falling back to public interface 2024-09-17T19:10:40.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:40.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:40.992 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:10:40.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:10:40.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:40.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:42.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:42.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:42.309 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:10:42.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:10:42.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:42.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:42.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:43.294 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:10:43.293+0000 7f132c15a7c0 -1 osd.3 0 log_to_monitors true 2024-09-17T19:10:43.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:43.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:43.639 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:10:43.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:10:43.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:43.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:44.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:45.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:45.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:45.094 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:10:45.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:10:45.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:45.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:45.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:46.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:46.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:46.437 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:10:46.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:10:46.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:46.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:46.769 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:47.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:47.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:47.771 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:10:47.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:10:47.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:47.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:48.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:49.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:49.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:49.104 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:10:49.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:10:49.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:49.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:49.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:50.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:50.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:50.438 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:10:50.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:10:50.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:50.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:50.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:51.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:51.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:51.775 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:10:51.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:10:51.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:51.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:52.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:53.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:53.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:53.262 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:10:53.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:10:53.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:53.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:53.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:54.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:54.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:54.587 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:10:54.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:10:54.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:54.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:54.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:55.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:55.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:55.904 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:10:55.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:10:55.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:55.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:56.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:57.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:57.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:57.242 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:10:57.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:10:57.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:57.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:57.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:10:58.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:10:58.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:10:58.994 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:10:58.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:10:58.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:10:58.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:10:59.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:00.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:00.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:00.751 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:11:00.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:11:00.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:00.752 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:11:01.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:02.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:02.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:02.507 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:11:02.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:11:02.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:02.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:11:03.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:04.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:04.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:04.133 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:11:04.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:11:04.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:04.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:11:04.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:05.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:05.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:05.692 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:11:05.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:11:05.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:05.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:11:06.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:07.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:07.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:07.727 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:11:07.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:11:07.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:07.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:11:08.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:09.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:09.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:09.130 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:11:09.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:11:09.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:09.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:11:09.457 INFO:tasks.workunit.client.0.smithi105.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/945142125,v1:127.0.0.1:6827/945142125] [v2:127.0.0.1:6828/945142125,v1:127.0.0.1:6829/945142125] exists,up f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:11:09.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:11:09.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:11:09.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:11:09.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:11:09.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-09-17T19:11:09.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:11:09.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:11:09.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:11:09.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:11:09.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:11:09.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:11:09.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2024-09-17T19:11:09.465 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:11:09.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2c138350-9353-4769-a9a0-db3851813303 2024-09-17T19:11:09.466 INFO:tasks.workunit.client.0.smithi105.stdout:add osd4 2c138350-9353-4769-a9a0-db3851813303 2024-09-17T19:11:09.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 2c138350-9353-4769-a9a0-db3851813303' 2024-09-17T19:11:09.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:11:09.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBN1OlmELToHBAAU/wlOeuzWj2thXn02QgGYg== 2024-09-17T19:11:09.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBN1OlmELToHBAAU/wlOeuzWj2thXn02QgGYg=="}' 2024-09-17T19:11:09.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2c138350-9353-4769-a9a0-db3851813303 -i td/osd-scrub-repair/4/new.json 2024-09-17T19:11:09.879 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:11:09.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2024-09-17T19:11:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBN1OlmELToHBAAU/wlOeuzWj2thXn02QgGYg== --osd-uuid 2c138350-9353-4769-a9a0-db3851813303 2024-09-17T19:11:09.933 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:09.932+0000 7fe847a8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:09.981 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:09.980+0000 7fe847a8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:09.983 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:09.980+0000 7fe847a8f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:09.983 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:09.980+0000 7fe847a8f7c0 -1 bdev(0x558b74a72800 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2024-09-17T19:11:09.984 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:09.980+0000 7fe847a8f7c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-09-17T19:11:16.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-09-17T19:11:16.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:11:16.677 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd4 key to auth repository 2024-09-17T19:11:16.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-17T19:11:16.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:11:17.091 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.4 2024-09-17T19:11:17.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-17T19:11:17.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:11:17.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:11:17.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:11:17.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:11:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:17.116+0000 7f3fd33ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:17.121 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:17.120+0000 7f3fd33ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:17.122 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:17.120+0000 7f3fd33ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:11:17.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:17.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:17.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:18.587 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:18.584+0000 7f3fd33ff7c0 -1 Falling back to public interface 2024-09-17T19:11:18.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:18.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:18.814 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:11:18.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:11:18.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:18.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:19.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:20.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:20.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:20.150 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:11:20.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:11:20.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:20.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:20.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:21.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:21.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:21.475 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:11:21.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:11:21.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:21.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:21.713 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:21.712+0000 7f3fd33ff7c0 -1 osd.4 0 log_to_monitors true 2024-09-17T19:11:21.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:22.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:22.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:22.806 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:11:22.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:11:22.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:22.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:23.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:24.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:24.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:24.134 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:11:24.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:11:24.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:24.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:24.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:25.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:25.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:25.474 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:11:25.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:11:25.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:25.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:25.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:26.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:26.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:26.901 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:11:26.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:11:26.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:26.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:27.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:28.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:28.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:28.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:11:28.230 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:11:28.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:28.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:28.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:29.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:29.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:29.683 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:11:29.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:11:29.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:29.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:30.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:31.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:31.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:31.044 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:11:31.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:11:31.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:31.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:31.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:32.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:32.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:32.451 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:11:32.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:11:32.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:32.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:32.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:33.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:33.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:33.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:11:33.864 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:11:33.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:33.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:34.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:35.374 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:11:35.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:35.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:35.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:11:35.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:35.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:36.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:37.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:37.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:37.062 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:11:37.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:11:37.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:37.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:37.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:38.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:38.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:38.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:11:38.784 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:11:38.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:38.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:39.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:40.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:40.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:40.272 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:11:40.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:11:40.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:40.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:40.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:41.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:41.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:41.838 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:11:41.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:11:41.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:41.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:42.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:43.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:43.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:43.221 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:11:43.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:11:43.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:43.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:43.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:44.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:44.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:44.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:11:44.824 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:11:44.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:44.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:45.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:46.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:46.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:46.388 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:11:46.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:11:46.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:46.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-17T19:11:46.710 INFO:tasks.workunit.client.0.smithi105.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3290710422,v1:127.0.0.1:6835/3290710422] [v2:127.0.0.1:6836/3290710422,v1:127.0.0.1:6837/3290710422] exists,up 2c138350-9353-4769-a9a0-db3851813303 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:11:46.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:11:46.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:11:46.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:11:46.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:11:46.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:11:46.718 INFO:tasks.workunit.client.0.smithi105.stdout:add osd5 be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:11:46.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 be2838cb-14ce-4bea-9d85-dd1337ab113a' 2024-09-17T19:11:46.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:11:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBy1Olm8m4XLBAAYLukvJQZS3ki4qF5ijzhIQ== 2024-09-17T19:11:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBy1Olm8m4XLBAAYLukvJQZS3ki4qF5ijzhIQ=="}' 2024-09-17T19:11:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be2838cb-14ce-4bea-9d85-dd1337ab113a -i td/osd-scrub-repair/5/new.json 2024-09-17T19:11:47.201 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:11:47.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2024-09-17T19:11:47.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBy1Olm8m4XLBAAYLukvJQZS3ki4qF5ijzhIQ== --osd-uuid be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:11:47.258 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:47.255+0000 7f487d54e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:47.261 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:47.259+0000 7f487d54e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:47.262 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:47.259+0000 7f487d54e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:47.262 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:47.259+0000 7f487d54e7c0 -1 bdev(0x560c35599c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2024-09-17T19:11:47.262 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:47.259+0000 7f487d54e7c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-09-17T19:11:53.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-09-17T19:11:53.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:11:53.647 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd5 key to auth repository 2024-09-17T19:11:53.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-17T19:11:53.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:11:54.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-17T19:11:54.046 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.5 2024-09-17T19:11:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:11:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:11:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:11:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:11:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:54.075+0000 7f4c83b847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:54.126 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:54.123+0000 7f4c83b847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:54.128 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:54.123+0000 7f4c83b847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:54.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:11:54.389 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:11:54.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:54.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:11:54.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:55.379+0000 7f4c83b847c0 -1 Falling back to public interface 2024-09-17T19:11:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:55.722 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:11:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:11:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:11:56.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:57.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:57.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:57.046 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:11:57.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:11:57.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:57.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:11:57.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:58.155 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:11:58.151+0000 7f4c83b847c0 -1 osd.5 0 log_to_monitors true 2024-09-17T19:11:58.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:58.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:58.385 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:11:58.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:11:58.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:58.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:11:58.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:11:59.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:11:59.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:11:59.726 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:11:59.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:11:59.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:11:59.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:00.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:01.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:01.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:01.132 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:12:01.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:12:01.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:01.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:01.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:02.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:02.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:02.470 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:12:02.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:12:02.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:02.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:02.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:03.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:03.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:03.819 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:12:03.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:12:03.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:03.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:04.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:05.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:05.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:05.338 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:12:05.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:12:05.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:05.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:05.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:06.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:06.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:06.673 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:12:06.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:12:06.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:06.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:07.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:08.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:08.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:08.010 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:12:08.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:12:08.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:08.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:08.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:09.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:09.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:09.348 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:12:09.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:12:09.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:09.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:09.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:10.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:10.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:10.680 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:12:10.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:12:10.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:10.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:11.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:12.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:12.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:12.010 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:12:12.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:12:12.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:12.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:12.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:13.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:13.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:13.715 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:12:13.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:12:13.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:13.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:14.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:15.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:15.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:15.491 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:12:15.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:12:15.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:15.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:16.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:16.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:16.926 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:12:16.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:12:16.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:16.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:17.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:18.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:18.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:18.764 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:12:18.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:12:18.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:18.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:19.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:20.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:20.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:20.319 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:12:20.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:12:20.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:20.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:21.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:22.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:22.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:22.120 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:12:22.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:12:22.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:22.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:22.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:23.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:23.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:23.462 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:12:23.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:12:23.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:23.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:12:23.842 INFO:tasks.workunit.client.0.smithi105.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1218571515,v1:127.0.0.1:6843/1218571515] [v2:127.0.0.1:6844/1218571515,v1:127.0.0.1:6845/1218571515] exists,up be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:12:23.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:12:23.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:12:23.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:12:23.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:12:23.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:12:23.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:12:23.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:12:23.850 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:12:23.850 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:12:23.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:12:23.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=92275ec5-cf2e-4927-8145-45914ece4f76 2024-09-17T19:12:23.853 INFO:tasks.workunit.client.0.smithi105.stdout:add osd6 92275ec5-cf2e-4927-8145-45914ece4f76 2024-09-17T19:12:23.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 92275ec5-cf2e-4927-8145-45914ece4f76' 2024-09-17T19:12:23.854 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:12:23.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCX1OlmeA8ENBAASHVw+qbJbm93YoT6oZHjBQ== 2024-09-17T19:12:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCX1OlmeA8ENBAASHVw+qbJbm93YoT6oZHjBQ=="}' 2024-09-17T19:12:23.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 92275ec5-cf2e-4927-8145-45914ece4f76 -i td/osd-scrub-repair/6/new.json 2024-09-17T19:12:24.307 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:12:24.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2024-09-17T19:12:24.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCX1OlmeA8ENBAASHVw+qbJbm93YoT6oZHjBQ== --osd-uuid 92275ec5-cf2e-4927-8145-45914ece4f76 2024-09-17T19:12:24.368 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:24.363+0000 7f7df7dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:12:24.370 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:24.367+0000 7f7df7dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:12:24.371 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:24.367+0000 7f7df7dae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:12:24.372 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:24.367+0000 7f7df7dae7c0 -1 bdev(0x5622cd40dc00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2024-09-17T19:12:24.372 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:24.367+0000 7f7df7dae7c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-09-17T19:12:30.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-09-17T19:12:30.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:12:30.533 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd6 key to auth repository 2024-09-17T19:12:30.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2024-09-17T19:12:30.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:12:30.944 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.6 2024-09-17T19:12:30.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2024-09-17T19:12:30.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:12:30.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:12:30.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:12:30.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:12:30.976 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:30.975+0000 7fe1a8b0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:12:30.979 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:30.975+0000 7fe1a8b0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:12:30.980 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:30.975+0000 7fe1a8b0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:12:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2024-09-17T19:12:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:12:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2024-09-17T19:12:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:12:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:12:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:31.294 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:12:31.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:12:31.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:31.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:31.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:32.495 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:32.491+0000 7fe1a8b0c7c0 -1 Falling back to public interface 2024-09-17T19:12:32.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:32.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:32.640 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:12:32.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:12:32.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:32.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:32.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:33.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:33.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:33.957 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:12:33.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:12:33.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:33.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:34.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:35.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:35.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:35.284 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:12:35.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:12:35.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:35.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:35.581 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:12:35.563+0000 7fe1a8b0c7c0 -1 osd.6 0 log_to_monitors true 2024-09-17T19:12:35.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:36.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:36.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:36.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:12:36.610 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:12:36.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:36.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:36.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:37.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:37.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:37.948 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:12:37.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:12:37.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:37.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:38.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:39.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:39.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:39.561 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:12:39.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:12:39.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:39.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:40.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:41.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:41.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:41.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:12:41.047 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:12:41.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:41.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:41.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:42.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:42.450 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:12:42.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:12:42.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:42.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:43.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:43.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:43.779 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:12:43.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:12:43.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:43.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:44.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:45.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:45.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:45.106 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:12:45.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:12:45.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:45.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:45.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:46.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:46.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:46.448 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:12:46.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:12:46.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:46.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:46.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:47.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:47.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:47.784 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:12:47.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:12:47.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:47.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:48.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:49.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:49.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:49.140 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:12:49.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:12:49.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:49.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:49.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:50.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:50.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:50.850 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:12:50.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:12:50.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:50.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:51.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:52.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:52.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:52.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:12:52.582 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:12:52.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:52.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:53.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:54.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:54.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:54.104 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:12:54.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:12:54.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:54.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:54.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:55.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:55.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:55.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:12:55.777 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:12:55.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:55.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:56.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:57.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:57.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:57.442 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:12:57.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:12:57.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:57.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:57.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:12:58.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:12:58.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:12:58.794 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:12:58.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:12:58.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:12:58.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:12:59.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:00.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:00.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:00.119 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:13:00.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:13:00.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:00.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:13:00.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:01.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:01.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:01.788 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T19:13:01.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T19:13:01.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:01.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2024-09-17T19:13:02.179 INFO:tasks.workunit.client.0.smithi105.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3016609463,v1:127.0.0.1:6851/3016609463] [v2:127.0.0.1:6852/3016609463,v1:127.0.0.1:6853/3016609463] exists,up 92275ec5-cf2e-4927-8145-45914ece4f76 2024-09-17T19:13:02.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:13:02.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:13:02.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:13:02.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:13:02.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-09-17T19:13:02.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:13:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:13:02.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:13:02.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2024-09-17T19:13:02.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:13:02.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a5c86a4b-70c5-4d97-b78f-0936ac571a70 2024-09-17T19:13:02.186 INFO:tasks.workunit.client.0.smithi105.stdout:add osd7 a5c86a4b-70c5-4d97-b78f-0936ac571a70 2024-09-17T19:13:02.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 a5c86a4b-70c5-4d97-b78f-0936ac571a70' 2024-09-17T19:13:02.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:13:02.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC+1OlmiuwqDBAAjYfAyw37m/9XAh9cBcCzxw== 2024-09-17T19:13:02.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC+1OlmiuwqDBAAjYfAyw37m/9XAh9cBcCzxw=="}' 2024-09-17T19:13:02.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a5c86a4b-70c5-4d97-b78f-0936ac571a70 -i td/osd-scrub-repair/7/new.json 2024-09-17T19:13:02.591 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:13:02.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2024-09-17T19:13:02.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+1OlmiuwqDBAAjYfAyw37m/9XAh9cBcCzxw== --osd-uuid a5c86a4b-70c5-4d97-b78f-0936ac571a70 2024-09-17T19:13:02.642 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:02.638+0000 7f65c60c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:02.692 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:02.690+0000 7f65c60c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:02.694 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:02.690+0000 7f65c60c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:02.695 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:02.690+0000 7f65c60c37c0 -1 bdev(0x5645b237c800 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2024-09-17T19:13:02.695 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:02.690+0000 7f65c60c37c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-09-17T19:13:09.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-09-17T19:13:09.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:13:09.662 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd7 key to auth repository 2024-09-17T19:13:09.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2024-09-17T19:13:09.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:13:10.083 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.7 2024-09-17T19:13:10.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2024-09-17T19:13:10.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:13:10.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:13:10.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:13:10.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:13:10.116 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:10.114+0000 7f5aa88037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:10.144 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:10.142+0000 7f5aa88037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:10.146 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:10.142+0000 7f5aa88037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:10.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2024-09-17T19:13:10.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:13:10.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:10.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:10.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:11.419 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:11.414+0000 7f5aa88037c0 -1 Falling back to public interface 2024-09-17T19:13:11.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:11.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:11.750 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:13:11.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:13:11.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:11.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:12.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:13.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:13.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:13.065 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:13:13.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:13:13.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:13.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:13.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:14.200 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:14.194+0000 7f5aa88037c0 -1 osd.7 0 log_to_monitors true 2024-09-17T19:13:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:14.395 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:13:14.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:13:14.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:14.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:14.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:15.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:15.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:15.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:13:15.733 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:13:15.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:15.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:16.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:17.164 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:13:17.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:13:17.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:17.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:17.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:18.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:18.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:18.483 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:13:18.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:13:18.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:18.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:18.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:19.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:19.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:19.860 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:13:19.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:13:19.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:19.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:20.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:21.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:21.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:21.292 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:13:21.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:13:21.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:21.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:21.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:22.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:22.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:22.628 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:13:22.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:13:22.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:22.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:23.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:24.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:24.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:24.041 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:13:24.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:13:24.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:24.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:24.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:13:25.426 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:13:25.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:25.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:25.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:26.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:26.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:26.847 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:13:26.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:13:26.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:26.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:27.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:28.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:28.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:28.177 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:13:28.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:13:28.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:28.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:28.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:29.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:29.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:29.513 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:13:29.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:13:29.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:29.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:30.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:31.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:31.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:31.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:13:31.591 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:13:31.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:31.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:32.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:33.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:33.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:33.431 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:13:33.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:13:33.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:33.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:34.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:35.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:35.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:35.255 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:13:35.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:13:35.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:35.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:36.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:37.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:37.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:37.344 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:13:37.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:13:37.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:37.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:37.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:38.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:38.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:38.970 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:13:38.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:13:38.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:38.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:39.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:40.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:13:40.648 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:13:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:40.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2024-09-17T19:13:40.981 INFO:tasks.workunit.client.0.smithi105.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3261890901,v1:127.0.0.1:6859/3261890901] [v2:127.0.0.1:6860/3261890901,v1:127.0.0.1:6861/3261890901] exists,up a5c86a4b-70c5-4d97-b78f-0936ac571a70 2024-09-17T19:13:40.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:13:40.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:13:40.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:13:40.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:13:40.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-09-17T19:13:40.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:13:40.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:13:40.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:13:40.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:13:40.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:13:40.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:13:40.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:13:40.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:13:40.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:13:40.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a2e2d0c2-3c69-46a9-b5d2-cbb673ec4017 2024-09-17T19:13:40.989 INFO:tasks.workunit.client.0.smithi105.stdout:add osd8 a2e2d0c2-3c69-46a9-b5d2-cbb673ec4017 2024-09-17T19:13:40.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 a2e2d0c2-3c69-46a9-b5d2-cbb673ec4017' 2024-09-17T19:13:40.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:13:41.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDl1OlmfO2CABAAdl14xdd2qX70YeKZRVq6jw== 2024-09-17T19:13:41.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDl1OlmfO2CABAAdl14xdd2qX70YeKZRVq6jw=="}' 2024-09-17T19:13:41.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a2e2d0c2-3c69-46a9-b5d2-cbb673ec4017 -i td/osd-scrub-repair/8/new.json 2024-09-17T19:13:41.489 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:13:41.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2024-09-17T19:13:41.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDl1OlmfO2CABAAdl14xdd2qX70YeKZRVq6jw== --osd-uuid a2e2d0c2-3c69-46a9-b5d2-cbb673ec4017 2024-09-17T19:13:41.545 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:41.546+0000 7fa60b3497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:41.566 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:41.566+0000 7fa60b3497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:41.568 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:41.570+0000 7fa60b3497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:41.569 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:41.570+0000 7fa60b3497c0 -1 bdev(0x562ec0511c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2024-09-17T19:13:41.569 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:41.570+0000 7fa60b3497c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-09-17T19:13:48.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-09-17T19:13:48.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:13:48.055 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd8 key to auth repository 2024-09-17T19:13:48.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2024-09-17T19:13:48.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:13:48.464 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.8 2024-09-17T19:13:48.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2024-09-17T19:13:48.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:13:48.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:13:48.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:13:48.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:13:48.496 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:48.493+0000 7fc12ad237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:48.498 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:48.497+0000 7fc12ad237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:48.499 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:48.497+0000 7fc12ad237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:13:48.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2024-09-17T19:13:48.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:13:48.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2024-09-17T19:13:48.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:13:48.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:13:48.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:48.817 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:13:48.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:13:48.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:48.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:49.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:49.535 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:49.533+0000 7fc12ad237c0 -1 Falling back to public interface 2024-09-17T19:13:50.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:50.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:50.173 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:13:50.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:13:50.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:50.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:50.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:51.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:51.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:51.507 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:13:51.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:13:51.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:51.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:51.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:52.286 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:13:52.285+0000 7fc12ad237c0 -1 osd.8 0 log_to_monitors true 2024-09-17T19:13:52.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:52.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:52.836 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:13:52.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:13:52.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:52.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:53.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:54.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:54.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:54.177 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:13:54.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:13:54.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:54.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:54.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:55.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:55.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:55.515 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:13:55.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:13:55.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:55.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:55.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:56.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:56.873 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:13:56.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:56.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:13:56.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:56.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:57.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:58.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:58.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:58.355 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:13:58.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:13:58.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:58.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:13:58.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:13:59.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:13:59.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:13:59.690 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:13:59.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:13:59.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:13:59.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:00.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:01.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:01.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:01.075 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:14:01.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:14:01.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:01.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:01.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:02.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:02.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:02.475 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:14:02.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:14:02.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:02.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:02.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:03.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:03.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:03.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:14:03.811 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:14:03.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:03.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:04.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:05.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:05.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:05.318 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:14:05.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:14:05.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:05.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:05.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:06.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:06.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:06.654 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:14:06.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:14:06.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:06.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:06.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:07.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:07.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:07.987 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:14:07.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:14:07.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:07.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:08.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:09.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:09.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:09.498 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:14:09.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:14:09.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:09.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:10.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:11.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:11.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:11.358 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:14:11.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:14:11.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:11.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:12.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:13.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:13.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:13.252 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:14:13.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:14:13.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:13.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:13.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:14.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:14.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:14.594 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:14:14.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:14:14.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:14.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:15.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:16.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:16.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:16.186 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:14:16.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:14:16.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:16.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:16.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:17.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:17.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:17.656 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:14:17.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:14:17.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:17.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:18.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:19.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:19.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:19.157 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T19:14:19.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T19:14:19.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:19.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2024-09-17T19:14:19.514 INFO:tasks.workunit.client.0.smithi105.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/62241360,v1:127.0.0.1:6867/62241360] [v2:127.0.0.1:6868/62241360,v1:127.0.0.1:6869/62241360] exists,up a2e2d0c2-3c69-46a9-b5d2-cbb673ec4017 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2024-09-17T19:14:19.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:14:19.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:14:19.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:14:19.522 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:14:19.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:14:19.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:14:19.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:14:19.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:14:19.524 INFO:tasks.workunit.client.0.smithi105.stdout:add osd9 cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:14:19.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 cfb164f5-12b0-4235-8bdc-546f7d6208e8' 2024-09-17T19:14:19.524 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:14:19.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAL1elmYXBgIBAAhQBIBR7LxVYmAyJn0c70eQ== 2024-09-17T19:14:19.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAL1elmYXBgIBAAhQBIBR7LxVYmAyJn0c70eQ=="}' 2024-09-17T19:14:19.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cfb164f5-12b0-4235-8bdc-546f7d6208e8 -i td/osd-scrub-repair/9/new.json 2024-09-17T19:14:19.957 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:14:19.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2024-09-17T19:14:19.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAL1elmYXBgIBAAhQBIBR7LxVYmAyJn0c70eQ== --osd-uuid cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:14:20.007 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:20.005+0000 7f88e44ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:14:20.041 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:20.041+0000 7f88e44ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:14:20.042 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:20.041+0000 7f88e44ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:14:20.043 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:20.041+0000 7f88e44ac7c0 -1 bdev(0x5654b970bc00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2024-09-17T19:14:20.043 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:20.041+0000 7f88e44ac7c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-09-17T19:14:26.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-09-17T19:14:26.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:14:26.558 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd9 key to auth repository 2024-09-17T19:14:26.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2024-09-17T19:14:26.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:14:26.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2024-09-17T19:14:26.977 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.9 2024-09-17T19:14:26.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:14:26.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:14:26.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:14:26.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:14:27.009 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:27.009+0000 7f879ed817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:14:27.044 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:27.045+0000 7f879ed817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:14:27.046 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:27.045+0000 7f879ed817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:14:27.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2024-09-17T19:14:27.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:14:27.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:14:27.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:14:27.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:14:27.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:27.333 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:14:27.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:14:27.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:27.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:27.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:28.362 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:28.361+0000 7f879ed817c0 -1 Falling back to public interface 2024-09-17T19:14:28.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:28.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:28.647 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:14:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:14:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:29.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:29.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:29.984 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:14:29.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:14:29.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:29.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:30.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:31.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:31.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:31.320 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:14:31.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:14:31.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:31.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:31.474 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:14:31.473+0000 7f879ed817c0 -1 osd.9 0 log_to_monitors true 2024-09-17T19:14:31.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:32.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:32.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:32.693 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:14:32.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:14:32.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:32.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:33.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:34.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:34.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:34.030 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:14:34.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:14:34.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:34.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:34.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:35.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:35.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:35.368 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:14:35.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:14:35.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:35.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:35.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:36.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:36.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:36.850 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:14:36.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:14:36.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:36.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:37.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:38.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:38.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:38.186 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:14:38.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:14:38.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:38.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:38.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:39.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:39.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:39.693 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:14:39.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:14:39.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:39.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:40.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:41.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:41.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:41.176 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:14:41.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:14:41.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:41.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:41.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:42.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:42.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:42.649 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:14:42.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:14:42.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:42.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:42.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:43.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:43.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:43.978 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:14:43.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:14:43.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:43.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:44.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:45.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:45.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:45.433 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:14:45.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:14:45.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:45.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:45.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:46.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:46.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:46.914 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:14:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:14:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:46.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:47.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:48.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:48.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:48.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:14:48.747 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:14:48.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:48.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:49.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:50.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:50.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:50.540 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:14:50.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:14:50.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:50.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:51.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:52.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:52.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:52.215 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:14:52.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:14:52.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:52.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:52.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:53.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:53.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:53.711 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:14:53.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:14:53.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:53.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:54.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:55.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:55.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:55.556 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:14:55.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:14:55.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:55.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:56.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:14:57.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:14:57.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:14:57.092 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:14:57.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:14:57.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:14:57.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:14:57.453 INFO:tasks.workunit.client.0.smithi105.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2965042839,v1:127.0.0.1:6875/2965042839] [v2:127.0.0.1:6876/2965042839,v1:127.0.0.1:6877/2965042839] exists,up cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:14:57.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:14:57.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:14:57.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:14:57.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-09-17T19:14:57.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T19:14:57.788 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T19:14:57.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T19:14:57.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T19:14:58.340 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T19:14:58.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:14:59.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T19:15:03.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-09-17T19:15:03.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:15:03.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:15:03.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:15:03.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:15:03.137 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:15:03.137 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:15:03.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:15:03.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:15:03.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:15:03.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:15:03.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:15:03.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:15:03.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:15:03.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:15:03.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:15:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:15:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:15:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:15:03.677 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:03.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:15:03.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836552 2024-09-17T19:15:03.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836552 2024-09-17T19:15:03.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552' 2024-09-17T19:15:03.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:03.837 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:15:03.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673028 2024-09-17T19:15:03.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673028 2024-09-17T19:15:03.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028' 2024-09-17T19:15:03.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:03.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:15:04.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509504 2024-09-17T19:15:04.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509504 2024-09-17T19:15:04.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504' 2024-09-17T19:15:04.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:15:04.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345970 2024-09-17T19:15:04.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345970 2024-09-17T19:15:04.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970' 2024-09-17T19:15:04.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.249 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:15:04.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182442 2024-09-17T19:15:04.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182442 2024-09-17T19:15:04.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970 4-107374182442' 2024-09-17T19:15:04.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:15:04.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018918 2024-09-17T19:15:04.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018918 2024-09-17T19:15:04.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970 4-107374182442 5-128849018918' 2024-09-17T19:15:04.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:15:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855392 2024-09-17T19:15:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855392 2024-09-17T19:15:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970 4-107374182442 5-128849018918 6-150323855392' 2024-09-17T19:15:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:15:04.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691862 2024-09-17T19:15:04.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691862 2024-09-17T19:15:04.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970 4-107374182442 5-128849018918 6-150323855392 7-171798691862' 2024-09-17T19:15:04.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:15:04.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528343 2024-09-17T19:15:04.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528343 2024-09-17T19:15:04.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970 4-107374182442 5-128849018918 6-150323855392 7-171798691862 8-193273528343' 2024-09-17T19:15:04.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:04.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:15:05.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364813 2024-09-17T19:15:05.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364813 2024-09-17T19:15:05.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836552 1-42949673028 2-64424509504 3-85899345970 4-107374182442 5-128849018918 6-150323855392 7-171798691862 8-193273528343 9-214748364813' 2024-09-17T19:15:05.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:05.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836552 2024-09-17T19:15:05.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:05.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:15:05.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836552 2024-09-17T19:15:05.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:05.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836552 2024-09-17T19:15:05.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-09-17T19:15:05.100 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836552 2024-09-17T19:15:05.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:15:05.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836553 -lt 21474836552 2024-09-17T19:15:05.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:05.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673028 2024-09-17T19:15:05.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:05.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:15:05.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673028 2024-09-17T19:15:05.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:05.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673028 2024-09-17T19:15:05.426 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949673028 2024-09-17T19:15:05.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-09-17T19:15:05.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:15:05.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673028 -lt 42949673028 2024-09-17T19:15:05.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:05.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:05.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509504 2024-09-17T19:15:05.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:15:05.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509504 2024-09-17T19:15:05.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:05.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509504 2024-09-17T19:15:05.781 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509504 2024-09-17T19:15:05.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509504' 2024-09-17T19:15:05.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:15:06.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509504 -lt 64424509504 2024-09-17T19:15:06.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:06.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345970 2024-09-17T19:15:06.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:06.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:15:06.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345970 2024-09-17T19:15:06.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:06.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345970 2024-09-17T19:15:06.118 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345970 2024-09-17T19:15:06.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2024-09-17T19:15:06.118 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:15:06.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345970 -lt 85899345970 2024-09-17T19:15:06.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:06.457 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:06.457 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182442 2024-09-17T19:15:06.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:15:06.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:06.459 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182442 2024-09-17T19:15:06.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182442 2024-09-17T19:15:06.461 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.4 seq 107374182442 2024-09-17T19:15:06.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-09-17T19:15:06.461 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:15:06.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182442 -lt 107374182442 2024-09-17T19:15:06.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:06.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018918 2024-09-17T19:15:06.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:06.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:15:06.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018918 2024-09-17T19:15:06.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:06.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018918 2024-09-17T19:15:06.800 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.5 seq 128849018918 2024-09-17T19:15:06.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-09-17T19:15:06.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:15:07.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018918 -lt 128849018918 2024-09-17T19:15:07.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:07.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855392 2024-09-17T19:15:07.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:07.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:15:07.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855392 2024-09-17T19:15:07.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:07.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855392 2024-09-17T19:15:07.137 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.6 seq 150323855392 2024-09-17T19:15:07.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855392' 2024-09-17T19:15:07.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:15:07.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855393 -lt 150323855392 2024-09-17T19:15:07.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:07.500 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691862 2024-09-17T19:15:07.500 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:07.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:15:07.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691862 2024-09-17T19:15:07.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:07.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691862 2024-09-17T19:15:07.505 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.7 seq 171798691862 2024-09-17T19:15:07.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-09-17T19:15:07.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:15:07.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691863 -lt 171798691862 2024-09-17T19:15:07.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:07.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528343 2024-09-17T19:15:07.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:07.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:15:07.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528343 2024-09-17T19:15:07.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:07.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528343 2024-09-17T19:15:07.877 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.8 seq 193273528343 2024-09-17T19:15:07.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528343' 2024-09-17T19:15:07.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:15:08.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528343 -lt 193273528343 2024-09-17T19:15:08.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:08.271 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364813 2024-09-17T19:15:08.272 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:08.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:15:08.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364813 2024-09-17T19:15:08.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:08.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364813 2024-09-17T19:15:08.276 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.9 seq 214748364813 2024-09-17T19:15:08.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-09-17T19:15:08.276 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:15:08.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364814 -lt 214748364813 2024-09-17T19:15:08.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:15:08.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:08.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:09.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:15:09.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:15:09.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:15:09.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:15:09.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:15:09.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:15:09.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:15:09.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:15:09.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:15:09.386 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:15:09.386 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:09.386 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:09.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T19:15:09.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-09-17T19:15:10.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T19:15:10.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T19:15:10.924 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T19:15:10.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:15:11.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-17T19:15:11.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-17T19:15:12.474 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 2 allow_ec_overwrites to true 2024-09-17T19:15:12.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T19:15:12.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:15:12.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:15:12.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:15:12.496 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:15:12.496 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:15:12.496 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:15:12.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:15:12.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:15:12.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:15:12.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:15:12.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:15:12.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:15:12.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:15:12.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:15:12.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:15:13.203 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:15:13.204 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:15:13.204 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:15:13.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:15:13.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:13.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:15:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836555 2024-09-17T19:15:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836555 2024-09-17T19:15:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555' 2024-09-17T19:15:13.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:13.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:15:13.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673031 2024-09-17T19:15:13.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673031 2024-09-17T19:15:13.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031' 2024-09-17T19:15:13.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:13.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:15:13.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509507 2024-09-17T19:15:13.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509507 2024-09-17T19:15:13.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507' 2024-09-17T19:15:13.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:13.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:15:13.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345973 2024-09-17T19:15:13.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345973 2024-09-17T19:15:13.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973' 2024-09-17T19:15:13.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:13.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:15:14.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182445 2024-09-17T19:15:14.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182445 2024-09-17T19:15:14.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973 4-107374182445' 2024-09-17T19:15:14.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:14.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:15:14.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018921 2024-09-17T19:15:14.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018921 2024-09-17T19:15:14.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973 4-107374182445 5-128849018921' 2024-09-17T19:15:14.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:14.250 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:15:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855395 2024-09-17T19:15:14.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855395 2024-09-17T19:15:14.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973 4-107374182445 5-128849018921 6-150323855395' 2024-09-17T19:15:14.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:14.395 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:15:14.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691865 2024-09-17T19:15:14.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691865 2024-09-17T19:15:14.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973 4-107374182445 5-128849018921 6-150323855395 7-171798691865' 2024-09-17T19:15:14.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:14.538 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:15:14.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528346 2024-09-17T19:15:14.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528346 2024-09-17T19:15:14.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973 4-107374182445 5-128849018921 6-150323855395 7-171798691865 8-193273528346' 2024-09-17T19:15:14.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:14.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:15:14.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364816 2024-09-17T19:15:14.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364816 2024-09-17T19:15:14.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836555 1-42949673031 2-64424509507 3-85899345973 4-107374182445 5-128849018921 6-150323855395 7-171798691865 8-193273528346 9-214748364816' 2024-09-17T19:15:14.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:14.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836555 2024-09-17T19:15:14.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:14.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:15:14.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836555 2024-09-17T19:15:14.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:14.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836555 2024-09-17T19:15:14.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-09-17T19:15:14.830 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836555 2024-09-17T19:15:14.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:15:15.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836554 -lt 21474836555 2024-09-17T19:15:15.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:15:16.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:15:16.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:15:16.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836556 -lt 21474836555 2024-09-17T19:15:16.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:16.475 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:16.475 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673031 2024-09-17T19:15:16.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:15:16.477 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673031 2024-09-17T19:15:16.477 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:16.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673031 2024-09-17T19:15:16.478 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949673031 2024-09-17T19:15:16.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-09-17T19:15:16.479 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:15:16.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673031 -lt 42949673031 2024-09-17T19:15:16.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:16.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509507 2024-09-17T19:15:16.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:16.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:15:16.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509507 2024-09-17T19:15:16.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:16.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509507 2024-09-17T19:15:16.825 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509507 2024-09-17T19:15:16.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-09-17T19:15:16.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:15:17.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509507 -lt 64424509507 2024-09-17T19:15:17.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:17.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345973 2024-09-17T19:15:17.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:17.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:15:17.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345973 2024-09-17T19:15:17.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:17.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345973 2024-09-17T19:15:17.176 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.3 seq 85899345973 2024-09-17T19:15:17.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345973' 2024-09-17T19:15:17.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:15:17.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345973 -lt 85899345973 2024-09-17T19:15:17.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:17.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182445 2024-09-17T19:15:17.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:17.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:15:17.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182445 2024-09-17T19:15:17.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:17.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182445 2024-09-17T19:15:17.511 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.4 seq 107374182445 2024-09-17T19:15:17.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-09-17T19:15:17.512 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:15:17.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182446 -lt 107374182445 2024-09-17T19:15:17.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:17.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018921 2024-09-17T19:15:17.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:17.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:15:17.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018921 2024-09-17T19:15:17.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:17.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018921 2024-09-17T19:15:17.878 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.5 seq 128849018921 2024-09-17T19:15:17.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018921' 2024-09-17T19:15:17.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:15:18.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018921 -lt 128849018921 2024-09-17T19:15:18.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:18.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855395 2024-09-17T19:15:18.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:18.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:15:18.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:18.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855395 2024-09-17T19:15:18.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855395 2024-09-17T19:15:18.213 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.6 seq 150323855395 2024-09-17T19:15:18.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-09-17T19:15:18.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:15:18.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855396 -lt 150323855395 2024-09-17T19:15:18.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:18.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:18.562 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691865 2024-09-17T19:15:18.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:15:18.565 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691865 2024-09-17T19:15:18.565 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:18.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691865 2024-09-17T19:15:18.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-09-17T19:15:18.566 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.7 seq 171798691865 2024-09-17T19:15:18.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:15:18.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691866 -lt 171798691865 2024-09-17T19:15:18.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:18.884 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528346 2024-09-17T19:15:18.884 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:18.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:15:18.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528346 2024-09-17T19:15:18.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:18.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528346 2024-09-17T19:15:18.888 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.8 seq 193273528346 2024-09-17T19:15:18.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528346' 2024-09-17T19:15:18.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:15:19.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528346 -lt 193273528346 2024-09-17T19:15:19.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:19.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364816 2024-09-17T19:15:19.232 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:19.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:15:19.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364816 2024-09-17T19:15:19.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:19.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364816 2024-09-17T19:15:19.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364816' 2024-09-17T19:15:19.236 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.9 seq 214748364816 2024-09-17T19:15:19.236 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:15:19.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364817 -lt 214748364816 2024-09-17T19:15:19.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:15:19.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:19.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:19.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:15:19.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:15:19.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:15:19.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:15:19.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:15:19.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:15:19.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:15:19.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:15:20.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:15:20.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:15:20.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:20.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:15:20.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:15:20.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:15:21.203 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:15:21.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:15:21.933 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:15:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:15:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:15:21.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T19:15:22.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-09-17T19:15:22.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2024-09-17T19:15:22.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T19:15:22.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:15:22.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T19:15:22.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-09-17T19:15:22.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-09-17T19:15:22.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-09-17T19:15:22.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2024-09-17T19:15:22.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2024-09-17T19:15:22.780 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:15:22.780 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:9 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr:7' 2024-09-17T19:15:23.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T19:15:23.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:15:23.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:15:23.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:15:24.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T19:15:26.100 INFO:tasks.workunit.client.0.smithi105.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T19:15:26.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:15:26.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:15:26.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:15:26.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:15:26.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:15:26.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:15:26.638 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:15:26.638 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:15:26.638 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:15:26.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:15:26.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:15:26.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:15:26.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T19:15:26.640 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.3 2024-09-17T19:15:26.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:15:26.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T19:15:26.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T19:15:26.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:15:26.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:15:26.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:15:26.674 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:26.672+0000 7fbd012cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:15:26.677 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:26.676+0000 7fbd012cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:15:26.679 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:26.676+0000 7fbd012cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:15:26.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:26.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:15:27.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:27.978 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:27.976+0000 7fbd012cc7c0 -1 Falling back to public interface 2024-09-17T19:15:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:28.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:15:28.310 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:28.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:28.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:15:28.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:29.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:29.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:29.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:15:29.647 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:29.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:29.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:15:29.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:30.678 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:30.676+0000 7fbd012cc7c0 -1 osd.3 69 log_to_monitors true 2024-09-17T19:15:30.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:30.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:30.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:15:30.982 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:15:30.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:30.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:15:31.324 INFO:tasks.workunit.client.0.smithi105.stderr:osd.3 up in weight 1 up_from 73 up_thru 64 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/2654952804,v1:127.0.0.1:6827/2654952804] [v2:127.0.0.1:6828/2654952804,v1:127.0.0.1:6829/2654952804] exists,up f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:15:31.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:15:31.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:15:31.326 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:15:31.326 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:15:31.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:15:31.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:15:31.326 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:15:31.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:15:31.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:15:31.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:15:31.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:15:31.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:15:31.429 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:15:31.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:15:31.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:15:31.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:31.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:15:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836560 2024-09-17T19:15:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836560 2024-09-17T19:15:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560' 2024-09-17T19:15:32.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:32.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:15:32.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673036 2024-09-17T19:15:32.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673036 2024-09-17T19:15:32.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036' 2024-09-17T19:15:32.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:32.398 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:15:32.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509512 2024-09-17T19:15:32.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509512 2024-09-17T19:15:32.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512' 2024-09-17T19:15:32.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:32.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:15:32.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-17T19:15:32.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-17T19:15:32.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610' 2024-09-17T19:15:32.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:32.687 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:15:32.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182450 2024-09-17T19:15:32.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182450 2024-09-17T19:15:32.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610 4-107374182450' 2024-09-17T19:15:32.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:32.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:15:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018925 2024-09-17T19:15:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018925 2024-09-17T19:15:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610 4-107374182450 5-128849018925' 2024-09-17T19:15:32.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:32.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:15:33.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855400 2024-09-17T19:15:33.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855400 2024-09-17T19:15:33.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610 4-107374182450 5-128849018925 6-150323855400' 2024-09-17T19:15:33.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:33.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:15:33.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691870 2024-09-17T19:15:33.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691870 2024-09-17T19:15:33.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610 4-107374182450 5-128849018925 6-150323855400 7-171798691870' 2024-09-17T19:15:33.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:33.296 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:15:33.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528350 2024-09-17T19:15:33.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528350 2024-09-17T19:15:33.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610 4-107374182450 5-128849018925 6-150323855400 7-171798691870 8-193273528350' 2024-09-17T19:15:33.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:33.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:15:33.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364821 2024-09-17T19:15:33.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364821 2024-09-17T19:15:33.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836560 1-42949673036 2-64424509512 3-313532612610 4-107374182450 5-128849018925 6-150323855400 7-171798691870 8-193273528350 9-214748364821' 2024-09-17T19:15:33.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:33.635 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:33.636 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836560 2024-09-17T19:15:33.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:15:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836560 2024-09-17T19:15:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836560 2024-09-17T19:15:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-09-17T19:15:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836560 2024-09-17T19:15:33.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:15:33.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836560 -lt 21474836560 2024-09-17T19:15:33.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:33.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673036 2024-09-17T19:15:33.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:33.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:15:33.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673036 2024-09-17T19:15:33.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:33.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673036 2024-09-17T19:15:33.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-09-17T19:15:33.975 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673036 2024-09-17T19:15:33.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:15:34.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673036 -lt 42949673036 2024-09-17T19:15:34.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:34.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509512 2024-09-17T19:15:34.336 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:34.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:15:34.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509512 2024-09-17T19:15:34.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:34.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509512 2024-09-17T19:15:34.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-09-17T19:15:34.340 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509512 2024-09-17T19:15:34.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:15:34.711 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509512 -lt 64424509512 2024-09-17T19:15:34.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:34.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612610 2024-09-17T19:15:34.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:34.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:15:34.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612610 2024-09-17T19:15:34.715 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-17T19:15:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-09-17T19:15:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 313532612610 2024-09-17T19:15:34.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:15:35.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-17T19:15:35.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:35.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182450 2024-09-17T19:15:35.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:35.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:15:35.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:35.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182450 2024-09-17T19:15:35.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182450 2024-09-17T19:15:35.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-09-17T19:15:35.051 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182450 2024-09-17T19:15:35.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:15:35.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182450 -lt 107374182450 2024-09-17T19:15:35.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:35.400 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018925 2024-09-17T19:15:35.400 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:35.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:15:35.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018925 2024-09-17T19:15:35.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:35.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018925 2024-09-17T19:15:35.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-09-17T19:15:35.404 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 128849018925 2024-09-17T19:15:35.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:15:35.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018926 -lt 128849018925 2024-09-17T19:15:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:35.736 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855400 2024-09-17T19:15:35.736 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:35.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:15:35.739 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855400 2024-09-17T19:15:35.739 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:35.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855400 2024-09-17T19:15:35.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-09-17T19:15:35.741 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855400 2024-09-17T19:15:35.741 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:15:36.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855400 -lt 150323855400 2024-09-17T19:15:36.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:36.076 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691870 2024-09-17T19:15:36.076 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:36.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:15:36.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691870 2024-09-17T19:15:36.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:36.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691870 2024-09-17T19:15:36.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-09-17T19:15:36.081 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691870 2024-09-17T19:15:36.081 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:15:36.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691870 -lt 171798691870 2024-09-17T19:15:36.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:36.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528350 2024-09-17T19:15:36.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:36.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:15:36.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528350 2024-09-17T19:15:36.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:36.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528350 2024-09-17T19:15:36.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-09-17T19:15:36.414 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528350 2024-09-17T19:15:36.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:15:36.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528351 -lt 193273528350 2024-09-17T19:15:36.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:36.741 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-214748364821 2024-09-17T19:15:36.741 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:36.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:15:36.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-214748364821 2024-09-17T19:15:36.745 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:36.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364821 2024-09-17T19:15:36.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 214748364821' 2024-09-17T19:15:36.746 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 214748364821 2024-09-17T19:15:36.746 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:15:37.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364821 -lt 214748364821 2024-09-17T19:15:37.081 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:15:37.081 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:37.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:37.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:15:37.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:15:37.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:15:37.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:15:37.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:15:37.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:15:37.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:15:37.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:15:37.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:15:37.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:15:37.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:37.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:38.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:15:38.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:15:38.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:15:38.236 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T19:15:38.236 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:15:38.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:15:38.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:15:38.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:15:38.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T19:15:38.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T19:15:38.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:15:38.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:15:38.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:15:38.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:15:38.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:15:38.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:15:38.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:15:10.799481+0000 2024-09-17T19:15:38.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:15:39.227 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:15:39.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:15:10.799481+0000 2024-09-17T19:15:39.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:15:39.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:15:10.799481+0000 2024-09-17T19:15:39.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:15:39.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:15:39.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:15:39.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:15:39.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:15:39.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:15:39.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:15:39.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:15:39.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:15:10.799481+0000 '>' 2024-09-17T19:15:10.799481+0000 2024-09-17T19:15:39.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:15:40.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:15:40.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:15:40.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:15:40.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:15:40.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:15:40.550 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:15:40.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:15:40.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:15:10.799481+0000 '>' 2024-09-17T19:15:10.799481+0000 2024-09-17T19:15:40.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:15:41.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:15:41.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:15:41.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:15:41.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:15:41.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:15:41.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:15:41.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:15:42.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:15:39.691697+0000 '>' 2024-09-17T19:15:10.799481+0000 2024-09-17T19:15:42.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:15:42.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:15:44.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:15:44.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:15:44.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:15:44.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:15:44.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:15:44.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:15:44.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:15:44.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:15:44.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:15:44.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:15:44.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:15:44.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:15:44.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:15:44.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:15:44.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-17T19:15:45.189 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T19:15:45.189 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T19:15:45.189 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T19:15:45.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:15:45.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:15:45.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:15:45.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:15:45.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:15:45.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:15:45.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.9 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:15:45.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:15:45.514 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:45.511+0000 7f6b202217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:15:45.517 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:45.515+0000 7f6b202217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:15:45.518 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:45.515+0000 7f6b202217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:15:46.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:15:46.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:15:46.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:15:46.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:15:46.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:15:46.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:46.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:15:46.232 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:15:46.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:46.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:15:46.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:46.811 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:46.807+0000 7f6b202217c0 -1 Falling back to public interface 2024-09-17T19:15:47.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:47.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:47.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:15:47.629 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:47.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:47.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:15:47.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:15:48.959 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:48.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:48.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:15:49.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:50.170 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:15:50.167+0000 7f6b202217c0 -1 osd.9 74 log_to_monitors true 2024-09-17T19:15:50.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:50.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:50.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:15:50.296 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:15:50.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:50.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:15:50.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:15:51.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:15:51.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:15:51.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:15:51.631 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:15:51.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:15:51.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:osd.9 up in weight 1 up_from 79 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/163711494,v1:127.0.0.1:6875/163711494] [v2:127.0.0.1:6876/163711494,v1:127.0.0.1:6877/163711494] exists,up cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:15:51.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:15:51.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:15:51.969 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:15:51.969 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:15:51.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:15:51.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:15:51.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:15:52.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:15:52.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:15:52.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:15:52.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:15:52.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:15:52.085 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:15:52.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:15:52.420 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:15:52.420 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:15:52.420 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:15:52.420 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:15:52.420 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:52.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:15:52.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836565 2024-09-17T19:15:52.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836565 2024-09-17T19:15:52.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565' 2024-09-17T19:15:52.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:52.773 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:15:52.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673041 2024-09-17T19:15:52.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673041 2024-09-17T19:15:52.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041' 2024-09-17T19:15:52.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:52.921 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:15:53.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509517 2024-09-17T19:15:53.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509517 2024-09-17T19:15:53.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517' 2024-09-17T19:15:53.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:53.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:15:53.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612615 2024-09-17T19:15:53.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612615 2024-09-17T19:15:53.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615' 2024-09-17T19:15:53.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:53.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:15:53.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182455 2024-09-17T19:15:53.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182455 2024-09-17T19:15:53.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615 4-107374182455' 2024-09-17T19:15:53.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:53.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:15:53.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018930 2024-09-17T19:15:53.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018930 2024-09-17T19:15:53.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615 4-107374182455 5-128849018930' 2024-09-17T19:15:53.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:53.615 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:15:53.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855405 2024-09-17T19:15:53.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855405 2024-09-17T19:15:53.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615 4-107374182455 5-128849018930 6-150323855405' 2024-09-17T19:15:53.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:53.758 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:15:53.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691875 2024-09-17T19:15:53.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691875 2024-09-17T19:15:53.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615 4-107374182455 5-128849018930 6-150323855405 7-171798691875' 2024-09-17T19:15:53.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:53.902 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:15:54.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528356 2024-09-17T19:15:54.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528356 2024-09-17T19:15:54.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615 4-107374182455 5-128849018930 6-150323855405 7-171798691875 8-193273528356' 2024-09-17T19:15:54.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:15:54.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:15:54.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416386 2024-09-17T19:15:54.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416386 2024-09-17T19:15:54.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836565 1-42949673041 2-64424509517 3-313532612615 4-107374182455 5-128849018930 6-150323855405 7-171798691875 8-193273528356 9-339302416386' 2024-09-17T19:15:54.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:54.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836565 2024-09-17T19:15:54.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:54.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:15:54.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836565 2024-09-17T19:15:54.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:54.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836565 2024-09-17T19:15:54.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-09-17T19:15:54.188 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836565 2024-09-17T19:15:54.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:15:54.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836565 -lt 21474836565 2024-09-17T19:15:54.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:54.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673041 2024-09-17T19:15:54.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:54.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:15:54.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673041 2024-09-17T19:15:54.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:54.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673041 2024-09-17T19:15:54.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-09-17T19:15:54.499 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673041 2024-09-17T19:15:54.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:15:54.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673041 -lt 42949673041 2024-09-17T19:15:54.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:54.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509517 2024-09-17T19:15:54.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:54.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:15:54.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509517 2024-09-17T19:15:54.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:54.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509517 2024-09-17T19:15:54.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-09-17T19:15:54.887 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509517 2024-09-17T19:15:54.887 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:15:55.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509517 -lt 64424509517 2024-09-17T19:15:55.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:55.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612615 2024-09-17T19:15:55.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:55.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:15:55.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612615 2024-09-17T19:15:55.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:55.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612615 2024-09-17T19:15:55.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-09-17T19:15:55.279 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 313532612615 2024-09-17T19:15:55.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:15:55.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612615 -lt 313532612615 2024-09-17T19:15:55.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:55.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182455 2024-09-17T19:15:55.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:55.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:15:55.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182455 2024-09-17T19:15:55.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:55.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182455 2024-09-17T19:15:55.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2024-09-17T19:15:55.686 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182455 2024-09-17T19:15:55.686 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:15:56.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182455 -lt 107374182455 2024-09-17T19:15:56.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:56.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:56.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018930 2024-09-17T19:15:56.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:15:56.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018930 2024-09-17T19:15:56.072 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:56.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018930 2024-09-17T19:15:56.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018930' 2024-09-17T19:15:56.074 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 128849018930 2024-09-17T19:15:56.074 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:15:56.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018931 -lt 128849018930 2024-09-17T19:15:56.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:56.446 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855405 2024-09-17T19:15:56.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:56.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:15:56.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855405 2024-09-17T19:15:56.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:56.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855405 2024-09-17T19:15:56.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-09-17T19:15:56.451 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855405 2024-09-17T19:15:56.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:15:56.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855405 -lt 150323855405 2024-09-17T19:15:56.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:56.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691875 2024-09-17T19:15:56.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:56.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:15:56.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691875 2024-09-17T19:15:56.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:56.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691875 2024-09-17T19:15:56.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-09-17T19:15:56.810 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691875 2024-09-17T19:15:56.810 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:15:57.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691875 -lt 171798691875 2024-09-17T19:15:57.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:57.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528356 2024-09-17T19:15:57.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:57.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:15:57.157 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528356 2024-09-17T19:15:57.157 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:57.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528356 2024-09-17T19:15:57.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-09-17T19:15:57.159 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528356 2024-09-17T19:15:57.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:15:57.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528356 -lt 193273528356 2024-09-17T19:15:57.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:15:57.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-339302416386 2024-09-17T19:15:57.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:15:57.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:15:57.547 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-339302416386 2024-09-17T19:15:57.547 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:15:57.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416386 2024-09-17T19:15:57.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 339302416386' 2024-09-17T19:15:57.549 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 339302416386 2024-09-17T19:15:57.549 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:15:57.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416387 -lt 339302416386 2024-09-17T19:15:57.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:15:57.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:57.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:58.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:15:58.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:15:58.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:15:58.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:15:58.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:15:58.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:15:58.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:15:58.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:15:58.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:15:58.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:15:58.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:15:58.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:15:59.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:15:59.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:15:59.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:15:59.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:15:59.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:15:59.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:15:59.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:15:59.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:15:59.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:15:59.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-17T19:16:01.735 INFO:tasks.workunit.client.0.smithi105.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T19:16:02.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:16:02.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:16:02.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:16:02.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:16:02.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:16:02.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:16:02.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:16:02.267 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:16:02.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:16:02.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:16:02.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:16:02.269 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.5 2024-09-17T19:16:02.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:16:02.269 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:16:02.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:16:02.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:16:02.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:16:02.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:16:02.303 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:02.303+0000 7f051b06a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:02.306 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:02.303+0000 7f051b06a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:02.308 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:02.307+0000 7f051b06a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:16:02.609 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:16:02.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:02.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:02.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:03.626 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:03.623+0000 7f051b06a7c0 -1 Falling back to public interface 2024-09-17T19:16:03.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:03.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:03.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:16:03.966 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:16:03.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:03.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:04.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:05.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:05.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:05.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:16:05.286 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:16:05.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:05.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:05.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:06.336 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:06.335+0000 7f051b06a7c0 -1 osd.5 81 log_to_monitors true 2024-09-17T19:16:06.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:06.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:06.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:16:06.612 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:16:06.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:06.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:06.966 INFO:tasks.workunit.client.0.smithi105.stderr:osd.5 up in weight 1 up_from 85 up_thru 70 down_at 82 last_clean_interval [30,81) [v2:127.0.0.1:6842/3628346417,v1:127.0.0.1:6843/3628346417] [v2:127.0.0.1:6844/3628346417,v1:127.0.0.1:6845/3628346417] exists,up be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:16:06.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:16:06.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:16:06.969 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:16:06.969 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:16:06.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:16:06.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:16:06.969 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:16:07.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:16:07.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:16:07.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:16:07.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:16:07.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:16:07.072 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:16:07.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:16:07.652 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:16:07.652 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:16:07.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:07.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:16:07.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836569 2024-09-17T19:16:07.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836569 2024-09-17T19:16:07.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569' 2024-09-17T19:16:07.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:07.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:16:08.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673045 2024-09-17T19:16:08.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673045 2024-09-17T19:16:08.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045' 2024-09-17T19:16:08.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:08.181 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:16:08.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509521 2024-09-17T19:16:08.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509521 2024-09-17T19:16:08.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521' 2024-09-17T19:16:08.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:08.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:16:08.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612619 2024-09-17T19:16:08.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612619 2024-09-17T19:16:08.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619' 2024-09-17T19:16:08.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:08.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:16:09.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182459 2024-09-17T19:16:09.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182459 2024-09-17T19:16:09.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619 4-107374182459' 2024-09-17T19:16:09.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:09.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:16:09.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220162 2024-09-17T19:16:09.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220162 2024-09-17T19:16:09.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619 4-107374182459 5-365072220162' 2024-09-17T19:16:09.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:09.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:16:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855409 2024-09-17T19:16:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855409 2024-09-17T19:16:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619 4-107374182459 5-365072220162 6-150323855409' 2024-09-17T19:16:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:09.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:16:09.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691879 2024-09-17T19:16:09.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691879 2024-09-17T19:16:09.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619 4-107374182459 5-365072220162 6-150323855409 7-171798691879' 2024-09-17T19:16:09.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:09.625 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:16:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528360 2024-09-17T19:16:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528360 2024-09-17T19:16:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619 4-107374182459 5-365072220162 6-150323855409 7-171798691879 8-193273528360' 2024-09-17T19:16:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:09.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:16:09.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416390 2024-09-17T19:16:09.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416390 2024-09-17T19:16:09.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836569 1-42949673045 2-64424509521 3-313532612619 4-107374182459 5-365072220162 6-150323855409 7-171798691879 8-193273528360 9-339302416390' 2024-09-17T19:16:09.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:09.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836569 2024-09-17T19:16:09.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:09.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:16:09.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836569 2024-09-17T19:16:09.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:09.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836569 2024-09-17T19:16:09.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-09-17T19:16:09.917 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836569 2024-09-17T19:16:09.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:16:10.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836569 -lt 21474836569 2024-09-17T19:16:10.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:10.242 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:10.242 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673045 2024-09-17T19:16:10.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:16:10.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673045 2024-09-17T19:16:10.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:10.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673045 2024-09-17T19:16:10.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-09-17T19:16:10.246 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673045 2024-09-17T19:16:10.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:16:10.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673045 -lt 42949673045 2024-09-17T19:16:10.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:10.605 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509521 2024-09-17T19:16:10.605 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:10.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:16:10.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509521 2024-09-17T19:16:10.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:10.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509521 2024-09-17T19:16:10.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-09-17T19:16:10.610 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509521 2024-09-17T19:16:10.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:16:10.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509521 -lt 64424509521 2024-09-17T19:16:10.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:10.964 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:10.964 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612619 2024-09-17T19:16:10.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:16:10.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612619 2024-09-17T19:16:10.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:10.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612619 2024-09-17T19:16:10.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-09-17T19:16:10.968 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 313532612619 2024-09-17T19:16:10.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:16:11.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612619 -lt 313532612619 2024-09-17T19:16:11.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:11.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182459 2024-09-17T19:16:11.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:11.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:16:11.313 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182459 2024-09-17T19:16:11.314 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:11.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182459 2024-09-17T19:16:11.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-09-17T19:16:11.316 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182459 2024-09-17T19:16:11.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:16:11.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182459 -lt 107374182459 2024-09-17T19:16:11.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:11.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-365072220162 2024-09-17T19:16:11.667 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:11.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:16:11.668 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-365072220162 2024-09-17T19:16:11.668 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:11.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220162 2024-09-17T19:16:11.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 365072220162' 2024-09-17T19:16:11.670 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 365072220162 2024-09-17T19:16:11.670 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:16:12.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220162 -lt 365072220162 2024-09-17T19:16:12.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:12.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855409 2024-09-17T19:16:12.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:12.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:16:12.016 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855409 2024-09-17T19:16:12.016 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:12.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855409 2024-09-17T19:16:12.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-09-17T19:16:12.018 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855409 2024-09-17T19:16:12.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:16:12.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855409 -lt 150323855409 2024-09-17T19:16:12.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:12.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691879 2024-09-17T19:16:12.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:12.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:16:12.368 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691879 2024-09-17T19:16:12.368 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:12.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691879 2024-09-17T19:16:12.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-09-17T19:16:12.370 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691879 2024-09-17T19:16:12.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:16:12.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691880 -lt 171798691879 2024-09-17T19:16:12.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:12.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528360 2024-09-17T19:16:12.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:12.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:16:12.707 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528360 2024-09-17T19:16:12.707 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:12.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528360 2024-09-17T19:16:12.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-09-17T19:16:12.709 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528360 2024-09-17T19:16:12.709 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:16:13.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528360 -lt 193273528360 2024-09-17T19:16:13.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:13.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-339302416390 2024-09-17T19:16:13.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:13.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:16:13.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:13.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-339302416390 2024-09-17T19:16:13.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416390 2024-09-17T19:16:13.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 339302416390' 2024-09-17T19:16:13.054 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 339302416390 2024-09-17T19:16:13.054 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:16:13.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416391 -lt 339302416390 2024-09-17T19:16:13.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:16:13.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:13.394 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:16:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:16:14.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:16:14.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:16:14.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:14.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:14.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:16:14.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:16:14.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:16:14.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-09-17T19:16:14.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:16:14.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:16:14.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:16:14.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:16:14.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-09-17T19:16:14.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-09-17T19:16:14.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:16:14.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:16:14.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:14.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:14.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:14.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:15.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:15:39.691697+0000 2024-09-17T19:16:15.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:16:15.503 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:16:15.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:15:39.691697+0000 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:15:39.691697+0000 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:15.520 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:15.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:15.521 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:15.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:15:39.691697+0000 '>' 2024-09-17T19:15:39.691697+0000 2024-09-17T19:16:15.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:16:16.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:16:16.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:16:16.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:16:16.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:16.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:16.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:16.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:17.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:15.826510+0000 '>' 2024-09-17T19:15:39.691697+0000 2024-09-17T19:16:17.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:16:17.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:16:19.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:16:19.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:16:19.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:16:19.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:16:19.205 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:16:19.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:16:19.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:16:19.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:16:19.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:16:19.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:16:19.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:16:19.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:16:19.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:16:19.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:16:19.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:16:19.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-17T19:16:20.026 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T19:16:20.026 INFO:tasks.workunit.client.0.smithi105.stdout:hinfo_key 2024-09-17T19:16:20.026 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T19:16:20.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:16:20.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:16:20.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:16:20.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:16:20.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:16:20.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:16:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:16:20.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:16:20.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:16:20.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:16:20.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:16:20.336 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:16:20.336 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:16:20.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.9 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:16:20.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:16:20.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:16:20.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:16:20.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:16:20.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:16:20.370 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:20.367+0000 7f8cd4ffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:20.373 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:20.371+0000 7f8cd4ffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:20.374 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:20.371+0000 7f8cd4ffa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:20.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:16:20.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:16:20.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:16:20.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:16:20.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:16:20.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:20.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:16:20.863 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:16:20.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:20.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:21.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:21.650 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:21.647+0000 7f8cd4ffa7c0 -1 Falling back to public interface 2024-09-17T19:16:22.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:22.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:22.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:16:22.189 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:16:22.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:22.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:22.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:23.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:23.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:23.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:16:23.526 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:16:23.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:23.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:23.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:24.770 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:16:24.767+0000 7f8cd4ffa7c0 -1 osd.9 87 log_to_monitors true 2024-09-17T19:16:24.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:24.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:24.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:16:24.881 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:16:24.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:24.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:25.215 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:26.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:26.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:26.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:16:26.217 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:16:26.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:26.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:26.923 INFO:tasks.workunit.client.0.smithi105.stderr:osd.9 up in weight 1 up_from 91 up_thru 0 down_at 88 last_clean_interval [79,87) [v2:127.0.0.1:6874/1064275771,v1:127.0.0.1:6875/1064275771] [v2:127.0.0.1:6876/1064275771,v1:127.0.0.1:6877/1064275771] exists,up cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:16:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:16:26.925 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:16:26.926 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:16:26.926 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:16:26.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:16:26.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:16:26.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:16:27.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:16:27.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:16:27.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:16:27.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:16:27.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:16:27.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:16:27.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:5 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:6 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:7 2024-09-17T19:16:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:8 2024-09-17T19:16:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:9' 2024-09-17T19:16:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:16:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:27.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:16:27.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836574 2024-09-17T19:16:27.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836574 2024-09-17T19:16:27.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574' 2024-09-17T19:16:27.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:27.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:16:27.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673050 2024-09-17T19:16:27.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673050 2024-09-17T19:16:27.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050' 2024-09-17T19:16:27.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:27.820 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:16:27.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509526 2024-09-17T19:16:27.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509526 2024-09-17T19:16:27.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526' 2024-09-17T19:16:27.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:27.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:16:28.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612624 2024-09-17T19:16:28.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612624 2024-09-17T19:16:28.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624' 2024-09-17T19:16:28.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:28.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:16:28.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182464 2024-09-17T19:16:28.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182464 2024-09-17T19:16:28.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624 4-107374182464' 2024-09-17T19:16:28.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:28.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:16:28.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=365072220167 2024-09-17T19:16:28.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 365072220167 2024-09-17T19:16:28.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624 4-107374182464 5-365072220167' 2024-09-17T19:16:28.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:28.371 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:16:28.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855414 2024-09-17T19:16:28.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855414 2024-09-17T19:16:28.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624 4-107374182464 5-365072220167 6-150323855414' 2024-09-17T19:16:28.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:28.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:16:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691884 2024-09-17T19:16:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691884 2024-09-17T19:16:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624 4-107374182464 5-365072220167 6-150323855414 7-171798691884' 2024-09-17T19:16:28.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:28.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:16:28.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528364 2024-09-17T19:16:28.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528364 2024-09-17T19:16:28.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624 4-107374182464 5-365072220167 6-150323855414 7-171798691884 8-193273528364' 2024-09-17T19:16:28.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:28.789 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:16:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-17T19:16:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-17T19:16:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836574 1-42949673050 2-64424509526 3-313532612624 4-107374182464 5-365072220167 6-150323855414 7-171798691884 8-193273528364 9-390842023938' 2024-09-17T19:16:28.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:28.923 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836574 2024-09-17T19:16:28.923 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:28.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:16:28.924 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836574 2024-09-17T19:16:28.924 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:28.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836574 2024-09-17T19:16:28.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2024-09-17T19:16:28.925 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836574 2024-09-17T19:16:28.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:16:29.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836573 -lt 21474836574 2024-09-17T19:16:29.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:16:30.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:16:30.265 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:16:30.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836574 -lt 21474836574 2024-09-17T19:16:30.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:30.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673050 2024-09-17T19:16:30.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:30.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:16:30.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673050 2024-09-17T19:16:30.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:30.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673050 2024-09-17T19:16:30.597 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-17T19:16:30.597 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949673050 2024-09-17T19:16:30.597 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:16:30.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-17T19:16:30.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:30.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509526 2024-09-17T19:16:30.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:30.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:16:30.952 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:30.952 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509526 2024-09-17T19:16:30.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509526 2024-09-17T19:16:30.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-09-17T19:16:30.953 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509526 2024-09-17T19:16:30.954 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:16:31.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509526 -lt 64424509526 2024-09-17T19:16:31.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:31.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612624 2024-09-17T19:16:31.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:31.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:16:31.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612624 2024-09-17T19:16:31.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612624 2024-09-17T19:16:31.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-09-17T19:16:31.294 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.3 seq 313532612624 2024-09-17T19:16:31.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:16:31.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612625 -lt 313532612624 2024-09-17T19:16:31.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:31.672 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182464 2024-09-17T19:16:31.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:31.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:16:31.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182464 2024-09-17T19:16:31.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:31.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182464 2024-09-17T19:16:31.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-09-17T19:16:31.676 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.4 seq 107374182464 2024-09-17T19:16:31.677 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:16:31.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182464 -lt 107374182464 2024-09-17T19:16:31.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:32.000 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-365072220167 2024-09-17T19:16:32.000 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:32.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:16:32.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-365072220167 2024-09-17T19:16:32.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:32.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=365072220167 2024-09-17T19:16:32.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 365072220167' 2024-09-17T19:16:32.004 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.5 seq 365072220167 2024-09-17T19:16:32.004 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:16:32.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 365072220167 -lt 365072220167 2024-09-17T19:16:32.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:32.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855414 2024-09-17T19:16:32.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:32.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:16:32.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855414 2024-09-17T19:16:32.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:32.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855414 2024-09-17T19:16:32.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-09-17T19:16:32.331 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.6 seq 150323855414 2024-09-17T19:16:32.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:16:32.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855414 -lt 150323855414 2024-09-17T19:16:32.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:32.669 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691884 2024-09-17T19:16:32.669 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:32.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:16:32.672 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691884 2024-09-17T19:16:32.672 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:32.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691884 2024-09-17T19:16:32.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-09-17T19:16:32.673 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.7 seq 171798691884 2024-09-17T19:16:32.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:16:33.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691885 -lt 171798691884 2024-09-17T19:16:33.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:33.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:33.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528364 2024-09-17T19:16:33.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:16:33.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:33.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528364 2024-09-17T19:16:33.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528364 2024-09-17T19:16:33.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528364' 2024-09-17T19:16:33.043 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.8 seq 193273528364 2024-09-17T19:16:33.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:16:33.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528365 -lt 193273528364 2024-09-17T19:16:33.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:33.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-390842023938 2024-09-17T19:16:33.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:33.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:16:33.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-390842023938 2024-09-17T19:16:33.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:33.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-17T19:16:33.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 390842023938' 2024-09-17T19:16:33.383 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.9 seq 390842023938 2024-09-17T19:16:33.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:16:33.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023939 -lt 390842023938 2024-09-17T19:16:33.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:16:33.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:33.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:34.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:16:34.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:16:34.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:16:34.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:16:34.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:16:34.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:16:34.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:16:34.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:16:34.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:16:34.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:16:34.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:34.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:34.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:16:34.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:16:34.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:16:34.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:16:34.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:16:34.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-09-17T19:16:34.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T19:16:34.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 244225"' 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 244225' 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:16:34.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 244226"' 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 244226' 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 244225 244226' 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 244225 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/244228: /' 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:16:34.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/244230: /' 2024-09-17T19:16:38.741 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.741 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:16:38.741 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:16:38.741 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:16:38.741 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:16:38.742 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:16:38.743 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.744 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:16:38.744 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:16:38.744 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:16:38.744 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:16:38.744 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:16:38.744 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:16:38.746 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:16:38.747 INFO:tasks.workunit.client.0.smithi105.stderr:244228: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:16:38.748 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:16:38.749 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:16:38.749 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:16:38.749 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:16:38.749 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:16:38.749 INFO:tasks.workunit.client.0.smithi105.stderr:244228: start osd.5 2024-09-17T19:16:38.749 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:16:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:2244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:16:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:16:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:16:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:16:38.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-09-17T19:16:38.800 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: remove 2#2:eb822e21:::SOMETHING:head# 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:16:38.801 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:16:38.802 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:16:38.803 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: start osd.9 2024-09-17T19:16:38.804 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2024-09-17T19:16:38.827+0000 7fbe6b3167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2024-09-17T19:16:38.831+0000 7fbe6b3167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2024-09-17T19:16:38.835+0000 7fbe6b3167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 0 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:43.570 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 1 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2024-09-17T19:16:40.599+0000 7fbe6b3167c0 -1 Falling back to public interface 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:16:43.571 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 3 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:43.572 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:44.181 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2024-09-17T19:16:43.567+0000 7fbe6b3167c0 -44228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:16:44.181 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:16:44.181 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:16:44.181 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2024-09-17T19:16:38.779+0000 7f94abccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2024-09-17T19:16:38.783+0000 7f94abccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2024-09-17T19:16:38.783+0000 7f94abccb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:16:44.182 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 0 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2024-09-17T19:16:40.327+0000 7f94abccb7c0 -1 Falling back to public interface 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:16:44.183 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 1 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:44.184 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 3 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:44.185 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:45.796 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2024-09-17T19:16:44.178+0000 7f94abccb7c0 -1 osd.9 93 log_to_monitors true 2024-09-17T19:16:45.796 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:45.796 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 4 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: osd.9 up in weight 1 up_from 98 up_thru 0 down_at 94 last_clean_interval [91,93) [v2:127.0.0.1:6874/1095295437,v1:127.0.0.1:6875/1095295437] [v2:127.0.0.1:6876/1095295437,v1:127.0.0.1:6877/1095295437] exists,up cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:16:45.797 INFO:tasks.workunit.client.0.smithi105.stderr:244230: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 1 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 2 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 3 2024-09-17T19:16:45.798 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 4 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 5 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 6 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 7 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 8 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: 9' 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836579 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836579 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579' 2024-09-17T19:16:45.799 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.244 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //1 osd.5 93 log_to_monitors true 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 4 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 5 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: osd.5 up in weight 1 up_from 100 up_thru 100 down_at 94 last_clean_interval [85,93) [v2:127.0.0.1:6842/199857375,v1:127.0.0.1:6843/199857375] [v2:127.0.0.1:6844/199857375,v1:127.0.0.1:6845/199857375] exists,up be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:16:47.245 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:16:47.246 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 1 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 2 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 3 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 4 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 5 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 6 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 7 2024-09-17T19:16:47.247 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 8 2024-09-17T19:16:47.248 INFO:tasks.workunit.client.0.smithi105.stderr:244228: 9' 2024-09-17T19:16:47.248 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:16:47.981 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:16:47.981 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673055 2024-09-17T19:16:47.981 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673055 2024-09-17T19:16:47.981 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055' 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509530 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509530 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530' 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612629 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612629 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629' 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182469 2024-09-17T19:16:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182469 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629 4-107374182469' 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729602 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729602 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629 4-107374182469 5-429496729602' 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855419 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855419 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629 4-107374182469 5-429496729602 6-150323855419' 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:16:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_psh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:16:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836580 2024-09-17T19:16:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836580 2024-09-17T19:16:48.544 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580' 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673056 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673056 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056' 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509532 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509532 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532' 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:16:48.545 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612630 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612630 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630' 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182470 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182470 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630 4-107374182470' 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729603 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729603 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630 4-107374182470 5-429496729603' 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:16:48.546 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855420 2024-09-17T19:16:48.600 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /hog_stats: seq=171798691889 2024-09-17T19:16:48.600 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691889 2024-09-17T19:16:48.600 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629 4-107374182469 5-429496729602 6-150323855419 7-171798691889' 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528369 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528369 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629 4-107374182469 5-429496729602 6-150323855419 7-171798691889 8-193273528369' 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836579 1-42949673055 2-64424509530 3-313532612629 4-107374182469 5-429496729602 6-150323855419 7-171798691889 8-193273528369 9-420906795010' 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836579 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:16:48.601 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836579 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836579 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.0 seq 21474836579 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836579 -lt 21474836579 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673055 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673055 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673055 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-09-17T19:16:48.602 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.1 seq 42949673055 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stme/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855420 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630 4-107374182470 5-429496729603 6-150323855420' 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691890 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691890 2024-09-17T19:16:49.314 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630 4-107374182470 5-429496729603 6-150323855420 7-171798691890' 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528370 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528370 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630 4-107374182470 5-429496729603 6-150323855420 7-171798691890 8-193273528370' 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795011 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795011 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836580 1-42949673056 2-64424509532 3-313532612630 4-107374182470 5-429496729603 6-150323855420 7-171798691890 8-193273528370 9-420906795011' 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836580 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:49.315 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836580 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836580 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.0 seq 21474836580 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836580 -lt 21474836580 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673056 2024-09-17T19:16:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.057 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helats: ceph osd last-stat-seq 1 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673055 -lt 42949673055 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509530 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509530 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509530 2024-09-17T19:16:50.058 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.2 seq 64424509530 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509530 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612629 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612629 2024-09-17T19:16:50.059 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612629 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612629' 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.3 seq 313532612629 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612630 -lt 313532612629 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182469 2024-09-17T19:16:50.060 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182469 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182469 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.4 seq 107374182469 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182470 -lt 107374182469 2024-09-17T19:16:50.061 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.062 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-429496729602 2024-09-17T19:16:50.326 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/ceppers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673056 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673056 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.1 seq 42949673056 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673056 -lt 42949673056 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.327 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509532 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509532 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509532 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.2 seq 64424509532 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:16:50.328 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509532 -lt 64424509532 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612630 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612630 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612630 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612630' 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.3 seq 313532612630 2024-09-17T19:16:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612630 -lt 313532612630 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182470 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182470 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:50.330 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182470 2024-09-17T19:16:51.095 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.clienhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.095 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:16:51.095 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-429496729602 2024-09-17T19:16:51.095 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729602 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 429496729602' 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.5 seq 429496729602 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729603 -lt 429496729602 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855419 2024-09-17T19:16:51.096 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855419 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855419 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.6 seq 150323855419 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:16:51.097 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855420 -lt 150323855419 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691889 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691889 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691889 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-09-17T19:16:51.098 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.7 seq 171798691889 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691890 -lt 171798691889 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528369 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:16:51.099 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528369 2024-09-17T19:16:51.637 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-09-17T19:16:51.637 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.4 seq 107374182470 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182470 -lt 107374182470 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-429496729603 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-429496729603 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:51.638 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729603 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 429496729603' 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.5 seq 429496729603 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729603 -lt 429496729603 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.639 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855420 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855420 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855420 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.6 seq 150323855420 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855420 -lt 150323855420 2024-09-17T19:16:51.640 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691890 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691890 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691890 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-09-17T19:16:51.641 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.7 seq 171798691890 2024-09-17T19:16:51.642 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:16:51.642 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691891 -lt 171798691890 2024-09-17T19:16:52.939 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/t -d - -f 2 2024-09-17T19:16:52.939 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528369 2024-09-17T19:16:52.939 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-09-17T19:16:52.939 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.8 seq 193273528369 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528371 -lt 193273528369 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-420906795010 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-420906795010 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 420906795010' 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: waiting osd.9 seq 420906795010 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:16:52.940 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795010 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:16:52.941 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:52.942 INFO:tasks.workunit.client.0.smithi105.stderr:244230: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:52.942 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:16:52.942 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:16:52.942 INFO:tasks.workunit.client.0.smithi105.stderr:244230: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:16:52.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528370 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528370 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528370 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528370' 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.8 seq 193273528370 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528371 -lt 193273528370 2024-09-17T19:16:53.076 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-420906795011 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-420906795011 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795011 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 420906795011' 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: waiting osd.9 seq 420906795011 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795011 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:53.077 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:16:53.078 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:16:53.449 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:244228: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:244228: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 244226 2024-09-17T19:16:53.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:16:53.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:16:53.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T19:16:53.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T19:16:53.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:16:53.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:16:53.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:53.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:53.778 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:53.778 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:54.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:16:15.826510+0000 2024-09-17T19:16:54.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:16:54.391 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:16:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:16:15.826510+0000 2024-09-17T19:16:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:16:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:16:15.826510+0000 2024-09-17T19:16:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:16:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:16:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:16:54.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:16:54.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:54.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:54.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:54.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:54.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:15.826510+0000 '>' 2024-09-17T19:16:15.826510+0000 2024-09-17T19:16:54.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:16:55.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:16:55.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:16:55.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:16:55.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:55.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:55.722 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:55.722 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:56.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:15.826510+0000 '>' 2024-09-17T19:16:15.826510+0000 2024-09-17T19:16:56.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:16:57.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:16:57.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:16:57.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:16:57.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:16:57.066 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:16:57.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:16:57.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:16:57.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:54.857848+0000 '>' 2024-09-17T19:16:15.826510+0000 2024-09-17T19:16:57.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:16:57.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:16:59.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T19:16:59.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:16:59.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:16:59.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 247674"' 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 247674' 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 247677"' 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/247676: /' 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 247677' 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T19:16:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 247674 247677' 2024-09-17T19:16:59.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:16:59.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:16:59.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 247674 2024-09-17T19:16:59.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:16:59.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/247679: /' 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.598 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2024-09-17T19:17:00.599 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: _ 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: hinfo_key 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: snapset 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:17:00.600 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:17:00.603 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:17:00.604 INFO:tasks.workunit.client.0.smithi105.stderr:247679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2024-09-17T19:17:00.605 INFO:tasks.workunit.client.0.smithi105.stderr:247679: start osd.9 2024-09-17T19:17:00.627 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.627 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:17:00.628 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: _ 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: hinfo_key 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: snapset 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:17:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:17:00.630 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:17:00.630 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:17:00.630 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:17:00.631 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:17:00.632 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:17:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:17:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:17:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:17:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:17:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:17:00.633 INFO:tasks.workunit.client.0.smithi105.stderr:247676: start osd.5 2024-09-17T19:17:06.193 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:17:06.193 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-09-17T19:17:06.193 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2024-09-17T19:17:06.193 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2024-09-17T19:17:00.638+0000 7fbd911007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2024-09-17T19:17:00.814+0000 7fbd911007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2024-09-17T19:17:00.818+0000 7fbd911007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 0 2024-09-17T19:17:06.194 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2024-09-17T19:17:02.158+0000 7fbd911007c0 -1 Falling back to public interface 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 1 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:17:06.195 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 3 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:17:06.196 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2024-09-17T19:17:06.174+0000 7fbdvery_ops 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:17:06.524 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2024-09-17T19:17:00.658+0000 7f83407887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2024-09-17T19:17:00.818+0000 7f83407887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2024-09-17T19:17:00.818+0000 7f83407887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 0 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.525 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2024-09-17T19:17:02.382+0000 7f83407887c0 -1 Falling back to public interface 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 1 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:06.526 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 3 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:06.527 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.clien911007c0 -1 osd.9 101 log_to_monitors true 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 4 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2024-09-17T19:17:07.758 INFO:tasks.workunit.client.0.smithi105.stderr:247679: osd.9 up in weight 1 up_from 105 up_thru 0 down_at 102 last_clean_interval [98,101) [v2:127.0.0.1:6842/4262068448,v1:127.0.0.1:6843/4262068448] [v2:127.0.0.1:6844/4262068448,v1:127.0.0.1:6845/4262068448] exists,up cfb164f5-12b0-4235-8bdc-546f7d6208e8 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:17:07.759 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 1 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 2 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 3 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 4 2024-09-17T19:17:07.760 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 5 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 6 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 7 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 8 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: 9' 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836585 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836585 2024-09-17T19:17:07.761 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585' 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $t.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 4 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:09.247 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2024-09-17T19:17:06.582+0000 7f83407887c0 -1 osd.5 101 log_to_monitors true 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 5 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: osd.5 up in weight 1 up_from 107 up_thru 100 down_at 102 last_clean_interval [100,101) [v2:127.0.0.1:6874/3657327611,v1:127.0.0.1:6875/3657327611] [v2:127.0.0.1:6876/3657327611,v1:127.0.0.1:6877/3657327611] exists,up be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:17:09.248 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:17:09.249 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 1 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 2 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 3 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 4 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 5 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 6 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 7 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 8 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: 9' 2024-09-17T19:17:09.250 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:17:10.157 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standaids 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673061 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673061 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061' 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509537 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509537 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537' 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612635 2024-09-17T19:17:10.158 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612635 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635' 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182475 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182475 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635 4-107374182475' 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500674 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500674 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635 4-107374182475 5-459561500674' 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855425 2024-09-17T19:17:10.159 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855425 2024-09-17T19:17:10.160 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635 4-107374182475 5-459561500674 6-150323855425' 2024-09-17T19:17:10.160 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.160 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shlone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836587 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836587 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587' 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673062 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673062 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062' 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.794 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509538 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509538 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538' 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612636 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612636 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636' 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182476 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182476 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636 4-107374182476' 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.795 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:17:10.796 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500675 2024-09-17T19:17:10.796 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500675 2024-09-17T19:17:10.796 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636 4-107374182476 5-459561500675' 2024-09-17T19:17:10.796 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.796 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323:2210: flush_pg_stats: seq=171798691895 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691895 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635 4-107374182475 5-459561500674 6-150323855425 7-171798691895' 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528376 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528376 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635 4-107374182475 5-459561500674 6-150323855425 7-171798691895 8-193273528376' 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:10.825 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509537 3-313532612635 4-107374182475 5-459561500674 6-150323855425 7-171798691895 8-193273528376 9-450971566082' 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836585 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836585 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836585 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.0 seq 21474836585 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-17T19:17:10.826 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673061 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673061 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673061 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-17T19:17:10.827 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.1 seq 42949673061 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222855426 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855426 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636 4-107374182476 5-459561500675 6-150323855426' 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691897 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691897 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636 4-107374182476 5-459561500675 6-150323855426 7-171798691897' 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:17:11.775 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528377 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528377 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636 4-107374182476 5-459561500675 6-150323855426 7-171798691897 8-193273528377' 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566083 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566083 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509538 3-313532612636 4-107374182476 5-459561500675 6-150323855426 7-171798691897 8-193273528377 9-450971566083' 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836587 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836587 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:11.776 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836587 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.0 seq 21474836587 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836587 -lt 21474836587 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673062 2024-09-17T19:17:11.777 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.260 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/s2: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673061 -lt 42949673061 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509537 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509537 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509537 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.2 seq 64424509537 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509538 -lt 64424509537 2024-09-17T19:17:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612635 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612635 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612635 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.3 seq 313532612635 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612637 -lt 313532612635 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182475 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.262 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182475 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182475 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.4 seq 107374182475 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182476 -lt 107374182475 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.263 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-459561500674 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //htandalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673062 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673062 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.1 seq 42949673062 2024-09-17T19:17:12.786 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673063 -lt 42949673062 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509538 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509538 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509538 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.2 seq 64424509538 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509538 -lt 64424509538 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.787 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-313532612636 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-313532612636 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612636 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 313532612636' 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.3 seq 313532612636 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612637 -lt 313532612636 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182476 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:17:12.788 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182476 2024-09-17T19:17:12.789 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:12.789 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182476 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/ceome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-459561500674 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500674 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 459561500674' 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.5 seq 459561500674 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500675 -lt 459561500674 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:13.302 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855425 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855425 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855425 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.6 seq 150323855425 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855426 -lt 150323855425 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691895 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:17:13.303 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691895 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691895 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.7 seq 171798691895 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691897 -lt 171798691895 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528376 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:17:13.304 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528376 2024-09-17T19:17:14.094 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flushphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.4 seq 107374182476 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182476 -lt 107374182476 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-459561500675 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-459561500675 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500675 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 459561500675' 2024-09-17T19:17:14.095 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.5 seq 459561500675 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500676 -lt 459561500675 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855426 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855426 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855426 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.6 seq 150323855426 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855427 -lt 150323855426 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:14.096 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691897 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691897 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691897 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.7 seq 171798691897 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:17:14.097 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691897 -lt 171798691897 2024-09-17T19:17:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:247_pg_stats: cut -d - -f 2 2024-09-17T19:17:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528376 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.8 seq 193273528376 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528377 -lt 193273528376 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566082 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566082 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: waiting osd.9 seq 450971566082 2024-09-17T19:17:15.175 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566084 -lt 450971566082 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:17:15.176 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:15.177 INFO:tasks.workunit.client.0.smithi105.stderr:247679: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:15.177 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:17:15.177 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:17:15.177 INFO:tasks.workunit.client.0.smithi105.stderr:247679: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:17:15.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528377 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528377 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528377 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.8 seq 193273528377 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528377 -lt 193273528377 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566083 2024-09-17T19:17:15.514 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566083 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566083 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: waiting osd.9 seq 450971566083 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566084 -lt 450971566083 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:17:15.515 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:17:15.516 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:17:15.516 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:17:15.516 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:17:15.516 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:17:15.516 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:247676: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:247676: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:17:15.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 247677 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:17:15.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:17:15.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 250636"' 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 250636' 2024-09-17T19:17:15.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 250638"' 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 250638' 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 250636 250638' 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/250639: /' 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 250636 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:17:15.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/250641: /' 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T19:17:19.493 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:17:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: remove 0#2:eb822e21:::SOMETHING:head# 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:17:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:17:19.498 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:17:19.499 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:17:19.500 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:17:19.500 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T19:17:19.500 INFO:tasks.workunit.client.0.smithi105.stderr:250639: start osd.3 2024-09-17T19:17:19.500 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:17:20.211 INFO:tasks.workunit.client.0.smithi105.stderr:2250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:17:20.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:17:20.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:17:20.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:17:20.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: remove 1#2:eb822e21:::SOMETHING:head# 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:17:20.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:17:20.215 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:17:20.216 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:17:20.217 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:17:20.217 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:17:20.217 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:17:20.217 INFO:tasks.workunit.client.0.smithi105.stderr:250641: start osd.5 2024-09-17T19:17:20.217 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:17:24.232 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2024-09-17T19:17:19.530+0000 7f4ae95e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2024-09-17T19:17:19.610+0000 7f4ae95e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2024-09-17T19:17:19.610+0000 7f4ae95e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:17:24.233 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 0 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2024-09-17T19:17:20.974+0000 7f4ae95e37c0 -1 Falling back to public interface 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 1 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:24.234 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 3 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:24.235 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2024-09-17T19:17:24.110+0000 7f4ae95e37c0 -1 osd.3 110 log_to_monitors true 2024-09-17T19:17:25.209 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper50641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2024-09-17T19:17:20.242+0000 7f5071dd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2024-09-17T19:17:20.266+0000 7f5071dd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2024-09-17T19:17:20.266+0000 7f5071dd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:17:25.210 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 0 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2024-09-17T19:17:21.606+0000 7f5071dd87c0 -1 Falling back to public interface 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:25.211 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 1 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:25.212 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:25.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:17:25.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 3 2024-09-17T19:17:25.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:25.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:25.213 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2024-09-17T19:17:25.118+0000 7f5071dd87c0 -1 osd.5 110 log_to_monitors true 2024-09-17T19:17:26.680 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:26.680 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:26.680 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:26.680 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:17:26.680 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 4 2024-09-17T19:17:26.680 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: osd.3 up in weight 1 up_from 114 up_thru 114 down_at 111 last_clean_interval [73,110) [v2:127.0.0.1:6826/2963094029,v1:127.0.0.1:6827/2963094029] [v2:127.0.0.1:6828/2963094029,v1:127.0.0.1:6829/2963094029] exists,up f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:17:26.681 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:17:26.682 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 1 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 2 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 3 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 4 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 5 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 6 2024-09-17T19:17:26.683 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 7 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 8 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: 9' 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836591 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836591 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591' 2024-09-17T19:17:26.684 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:27.387 INFO:tasks.workunit.client.0.smithi105.stderr:2506s.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:27.387 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 4 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: osd.5 up in weight 1 up_from 117 up_thru 107 down_at 111 last_clean_interval [107,110) [v2:127.0.0.1:6874/4114884717,v1:127.0.0.1:6875/4114884717] [v2:127.0.0.1:6876/4114884717,v1:127.0.0.1:6877/4114884717] exists,up be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:17:27.388 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:17:27.389 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 1 2024-09-17T19:17:27.390 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 2 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 3 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 4 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 5 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 6 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 7 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 8 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: 9' 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:27.391 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:17:27.392 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836592 2024-09-17T19:17:27.392 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836592 2024-09-17T19:17:27.392 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592' 2024-09-17T19:17:27.392 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:25039: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:17:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673067 2024-09-17T19:17:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673067 2024-09-17T19:17:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067' 2024-09-17T19:17:28.410 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509543 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509543 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543' 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271746 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271746 2024-09-17T19:17:28.411 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746' 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182481 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182481 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746 4-107374182481' 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173634 2024-09-17T19:17:28.412 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173634 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746 4-107374182481 5-502511173634' 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855431 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855431 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746 4-107374182481 5-502511173634 6-150323855431' 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.413 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:17:28.878 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: f641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:17:28.878 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673068 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673068 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068' 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509544 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509544 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544' 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271747 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271747 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747' 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.879 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182482 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182482 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747 4-107374182482' 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173635 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173635 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747 4-107374182482 5-502511173635' 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855432 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855432 2024-09-17T19:17:28.880 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747 4-107374182482 5-502511173635 6-150323855432' 2024-09-17T19:17:28.881 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:28.881 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:17:29.048 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: lush_pg_stats: seq=171798691901 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691901 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746 4-107374182481 5-502511173634 6-150323855431 7-171798691901' 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528381 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528381 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746 4-107374182481 5-502511173634 6-150323855431 7-171798691901 8-193273528381' 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:17:29.049 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566088 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566088 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836591 1-42949673067 2-64424509543 3-489626271746 4-107374182481 5-502511173634 6-150323855431 7-171798691901 8-193273528381 9-450971566088' 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836591 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836591 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836591 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.0 seq 21474836591 2024-09-17T19:17:29.050 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836591 -lt 21474836591 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673067 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673067 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673067 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2024-09-17T19:17:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.1 seq 42949673067 2024-09-17T19:17:29.475 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flushflush_pg_stats: seq=171798691902 2024-09-17T19:17:29.475 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691902 2024-09-17T19:17:29.475 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747 4-107374182482 5-502511173635 6-150323855432 7-171798691902' 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528383 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528383 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747 4-107374182482 5-502511173635 6-150323855432 7-171798691902 8-193273528383' 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566089 2024-09-17T19:17:29.476 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566089 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836592 1-42949673068 2-64424509544 3-489626271747 4-107374182482 5-502511173635 6-150323855432 7-171798691902 8-193273528383 9-450971566089' 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836592 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836592 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836592 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2024-09-17T19:17:29.477 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.0 seq 21474836592 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836592 -lt 21474836592 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673068 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673068 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:29.478 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673068 2024-09-17T19:17:29.479 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673068' 2024-09-17T19:17:29.479 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.1 seq 42949673068 2024-09-17T19:17:30.543 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flus_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:17:30.543 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673068 -lt 42949673067 2024-09-17T19:17:30.543 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.543 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509543 2024-09-17T19:17:30.543 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509543 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509543 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.2 seq 64424509543 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509544 -lt 64424509543 2024-09-17T19:17:30.544 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-489626271746 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-489626271746 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271746 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.3 seq 489626271746 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:17:30.545 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271747 -lt 489626271746 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182481 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182481 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182481 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-09-17T19:17:30.546 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.4 seq 107374182481 2024-09-17T19:17:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:17:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182482 -lt 107374182481 2024-09-17T19:17:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.547 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-502511173634 2024-09-17T19:17:30.732 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubunh_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:17:30.732 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673068 -lt 42949673068 2024-09-17T19:17:30.732 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.732 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509544 2024-09-17T19:17:30.732 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:30.732 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509544 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509544 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509544' 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.2 seq 64424509544 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509544 -lt 64424509544 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-489626271747 2024-09-17T19:17:30.733 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-489626271747 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271747 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.3 seq 489626271747 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271747 -lt 489626271747 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.734 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182482 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182482 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182482 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.4 seq 107374182482 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182482 -lt 107374182482 2024-09-17T19:17:30.735 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:30.736 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-502511173635 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubutu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-502511173634 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173634 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 502511173634' 2024-09-17T19:17:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.5 seq 502511173634 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173635 -lt 502511173634 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855431 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855431 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855431 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.6 seq 150323855431 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855432 -lt 150323855431 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691901 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:17:31.572 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691901 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691901 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.7 seq 171798691901 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691903 -lt 171798691901 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528381 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.573 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:17:31.574 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528381 2024-09-17T19:17:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_statntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-502511173635 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173635 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 502511173635' 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.5 seq 502511173635 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173635 -lt 502511173635 2024-09-17T19:17:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855432 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855432 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855432 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.6 seq 150323855432 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:17:31.760 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855432 -lt 150323855432 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691902 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691902 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691902 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-09-17T19:17:31.761 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.7 seq 171798691902 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691903 -lt 171798691902 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528383 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:17:31.762 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528383 2024-09-17T19:17:33.356 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stas: cut -d - -f 2 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528381 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.8 seq 193273528381 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528383 -lt 193273528381 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566088 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566088 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566088 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566088' 2024-09-17T19:17:33.357 INFO:tasks.workunit.client.0.smithi105.stderr:250639: waiting osd.9 seq 450971566088 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566089 -lt 450971566088 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:17:33.358 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:250639: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:250639: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:17:33.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 250638 2024-09-17T19:17:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:ts: cut -d - -f 2 2024-09-17T19:17:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528383 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528383' 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.8 seq 193273528383 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528383 -lt 193273528383 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566089 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566089 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566089 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566089' 2024-09-17T19:17:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:250641: waiting osd.9 seq 450971566089 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566089 -lt 450971566089 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:17:33.597 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:250641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-09-17T19:17:33.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:17:33.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-09-17T19:17:33.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:17:33.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:17:33.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:17:33.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2024-09-17T19:17:33.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-09-17T19:17:33.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-09-17T19:17:33.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=2.0 2024-09-17T19:17:33.946 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 2.0 2024-09-17T19:17:33.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:33.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:33.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:33.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:34.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:34.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 2.0 2024-09-17T19:17:34.561 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-09-17T19:17:34.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 2.0 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:34.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=2.0 2024-09-17T19:17:34.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:34.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:17:34.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:17:34.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:17:34.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:17:34.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:34.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:34.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:34.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:34.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:54.857848+0000 '>' 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:34.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:17:35.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:17:35.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:17:35.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:17:35.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:35.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:35.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:35.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:36.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:54.857848+0000 '>' 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:36.236 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:17:37.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:17:37.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:17:37.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:17:37.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:37.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:37.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:37.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:37.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:54.857848+0000 '>' 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:37.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:17:38.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:17:38.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:17:38.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:17:38.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:38.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:38.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:38.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:38.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:54.857848+0000 '>' 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:38.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:17:39.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:17:39.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:17:39.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:17:39.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:39.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:39.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:39.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:40.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:16:54.857848+0000 '>' 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:40.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:17:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:17:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:17:41.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-09-17T19:17:41.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2024-09-17T19:17:41.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:17:41.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:17:41.254 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-09-17T19:17:41.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:17:35.110097+0000 '>' 2024-09-17T19:16:54.857848+0000 2024-09-17T19:17:41.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:17:41.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:17:43.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-09-17T19:17:43.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:17:43.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:17:43.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 254136"' 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 254136' 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: run_in_background: local pid_variable=pids 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: run_in_background: shift 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: eval 'pids+=" 254138"' 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: run_in_background: pids+=' 254138' 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-09-17T19:17:43.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: wait_background: pids=' 254136 254138' 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: wait_background: return_code=0 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 254136 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/254139: /' 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:17:43.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: sed 's/^/254141: /' 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:17:45.806 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-09-17T19:17:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:254141: _ 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: hinfo_key 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: snapset 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:17:45.808 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:17:45.810 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:17:45.810 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:17:45.810 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-09-17T19:17:45.810 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-09-17T19:17:45.810 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:17:45.810 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:17:45.811 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2024-09-17T19:17:45.812 INFO:tasks.workunit.client.0.smithi105.stderr:254141: start osd.5 2024-09-17T19:17:45.858 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:17:45.859 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: _ 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: hinfo_key 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: snapset 2024-09-17T19:17:45.860 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-09-17T19:17:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:17:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:17:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-17T19:17:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:17:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-09-17T19:17:45.861 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:17:45.862 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:17:45.863 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:17:45.864 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:17:45.864 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-09-17T19:17:45.864 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-17T19:17:45.864 INFO:tasks.workunit.client.0.smithi105.stderr:254139: start osd.3 2024-09-17T19:17:50.747 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:17:50.747 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-09-17T19:17:50.747 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-17T19:17:50.747 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2024-09-17T19:17:45.893+0000 7f6b5ab7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2024-09-17T19:17:45.893+0000 7f6b5ab7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2024-09-17T19:17:45.897+0000 7f6b5ab7a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:17:50.748 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 0 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:50.749 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2024-09-17T19:17:47.193+0000 7f6b5ab7a7c0 -1 Falling back to public interface 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 1 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:50.750 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 3 2024-09-17T19:17:50.751 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:50.752 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:51.425 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.very_ops 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2024-09-17T19:17:45.849+0000 7f2140cae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2024-09-17T19:17:45.853+0000 7f2140cae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2024-09-17T19:17:45.853+0000 7f2140cae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2024-09-17T19:17:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 0 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2024-09-17T19:17:47.401+0000 7f2140cae7c0 -1 Falling back to public interface 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:51.427 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 1 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:51.428 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 3 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:51.429 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:54.291 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:54.291 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:54.291 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:17:54.291 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 4 2024-09-17T19:17:54.291 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:54.291 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2024-09-17T19:17:51.501+0000 7f2140cae7c0 -1 osd.5 118 log_to_monitors true 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 5 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: osd.5 up in weight 1 up_from 124 up_thru 117 down_at 119 last_clean_interval [117,118) [v2:127.0.0.1:6874/3981314730,v1:127.0.0.1:6875/3981314730] [v2:127.0.0.1:6876/3981314730,v1:127.0.0.1:6877/3981314730] exists,up be2838cb-14ce-4bea-9d85-dd1337ab113a 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:17:54.292 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:17:54.293 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 1 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 2 2024-09-17T19:17:54.294 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 3 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 4 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 5 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 6 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 7 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 8 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: 9' 2024-09-17T19:17:54.295 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:17:54.526 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standa0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2024-09-17T19:17:51.417+0000 7f6b5ab7a7c0 -1 osd.3 118 log_to_monitors true 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 4 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:17:54.527 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 5 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: osd.3 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [114,118) [v2:127.0.0.1:6826/2553447467,v1:127.0.0.1:6827/2553447467] [v2:127.0.0.1:6828/2553447467,v1:127.0.0.1:6829/2553447467] exists,up f5ff2c0d-ce54-4109-a18b-7f2affb0093a 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:17:54.528 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:17:54.529 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:17:54.530 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 1 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 2 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 3 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 4 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 5 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 6 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 7 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 8 2024-09-17T19:17:54.531 INFO:tasks.workunit.client.0.smithi105.stderr:254139: 9' 2024-09-17T19:17:55.882 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalonlone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.882 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836599 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836599 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599' 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673074 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673074 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074' 2024-09-17T19:17:55.883 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509550 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509550 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550' 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410' 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.884 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182488 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182488 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410 4-107374182488' 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944706 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944706 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410 4-107374182488 5-532575944706' 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.885 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323e/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836600 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836600 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600' 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.899 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673075 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673075 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075' 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509551 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509551 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551' 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977411 2024-09-17T19:17:55.900 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977411 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411' 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182489 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182489 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411 4-107374182489' 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=532575944707 2024-09-17T19:17:55.901 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 532575944707 2024-09-17T19:17:55.902 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411 4-107374182489 5-532575944707' 2024-09-17T19:17:55.902 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:55.902 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-09-17T19:17:56.631 INFO:tasks.workunit.client.0.smithi105.stderr:254139:855438 2024-09-17T19:17:56.631 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855438 2024-09-17T19:17:56.631 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410 4-107374182488 5-532575944706 6-150323855438' 2024-09-17T19:17:56.631 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:56.631 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:17:56.631 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691909 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691909 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410 4-107374182488 5-532575944706 6-150323855438 7-171798691909' 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528389 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528389 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410 4-107374182488 5-532575944706 6-150323855438 7-171798691909 8-193273528389' 2024-09-17T19:17:56.632 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566095 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566095 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836599 1-42949673074 2-64424509550 3-528280977410 4-107374182488 5-532575944706 6-150323855438 7-171798691909 8-193273528389 9-450971566095' 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836599 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836599 2024-09-17T19:17:56.633 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836599 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.0 seq 21474836599 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836600 -lt 21474836599 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673074 2024-09-17T19:17:56.634 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/s /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=150323855439 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 150323855439 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411 4-107374182489 5-532575944707 6-150323855439' 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691910 2024-09-17T19:17:56.644 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691910 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411 4-107374182489 5-532575944707 6-150323855439 7-171798691910' 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528390 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528390 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411 4-107374182489 5-532575944707 6-150323855439 7-171798691910 8-193273528390' 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:17:56.645 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566096 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566096 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836600 1-42949673075 2-64424509551 3-528280977411 4-107374182489 5-532575944707 6-150323855439 7-171798691910 8-193273528390 9-450971566096' 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836600 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:17:56.646 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836600 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836600 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.0 seq 21474836600 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836600 -lt 21474836600 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:56.647 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673075 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ctandalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673074 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673074 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.1 seq 42949673074 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673075 -lt 42949673074 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:57.775 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509550 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509550 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509550 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509550' 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.2 seq 64424509550 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509552 -lt 64424509550 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-528280977410 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:17:57.776 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-528280977410 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 528280977410' 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.3 seq 528280977410 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977412 -lt 528280977410 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182488 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182488 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:57.777 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182488 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/ceeph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673075 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673075 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673075' 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.1 seq 42949673075 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673075 -lt 42949673075 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509551 2024-09-17T19:17:59.011 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509551 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509551 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509551' 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.2 seq 64424509551 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509550 -lt 64424509551 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509552 -lt 64424509551 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-528280977411 2024-09-17T19:17:59.012 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-528280977411 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977411 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 528280977411' 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.3 seq 528280977411 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977412 -lt 528280977411 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:59.013 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182489 2024-09-17T19:17:59.189 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182488' 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.4 seq 107374182488 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182490 -lt 107374182488 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-532575944706 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-532575944706 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944706 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 532575944706' 2024-09-17T19:17:59.190 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.5 seq 532575944706 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944707 -lt 532575944706 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855438 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855438 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855438 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855438' 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.6 seq 150323855438 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855440 -lt 150323855438 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:17:59.191 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691909 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691909 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691909 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691909' 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.7 seq 171798691909 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:17:59.192 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691910 -lt 171798691909 2024-09-17T19:18:00.019 INFO:tasks.workunit.client.0.smithi105.stderr:254/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:00.019 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-17T19:18:00.019 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182489 2024-09-17T19:18:00.019 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:00.019 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182489 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182489' 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.4 seq 107374182489 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182490 -lt 107374182489 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-532575944707 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-532575944707 2024-09-17T19:18:00.020 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=532575944707 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 532575944707' 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.5 seq 532575944707 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 532575944708 -lt 532575944707 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 6-150323855439 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=6 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 6-150323855439 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=150323855439 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.6 seq 150323855439' 2024-09-17T19:18:00.021 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.6 seq 150323855439 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 6 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 150323855440 -lt 150323855439 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 7-171798691910 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=7 2024-09-17T19:18:00.022 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 7-171798691910 2024-09-17T19:18:00.587 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flu141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528389 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528389 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528389 2024-09-17T19:18:00.588 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528389' 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.8 seq 193273528389 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528391 -lt 193273528389 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566095 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566095 2024-09-17T19:18:00.589 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566095 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566095' 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: waiting osd.9 seq 450971566095 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566097 -lt 450971566095 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:18:00.590 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:18:00.591 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:254141: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:254141: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:18:00.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:18:01.412 INFO:tasks.workunit.client.0.smithi105.stderr:sh_pg_stats: cut -d - -f 2 2024-09-17T19:18:01.412 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691910 2024-09-17T19:18:01.412 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.7 seq 171798691910' 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.7 seq 171798691910 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 7 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691910 -lt 171798691910 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 8-193273528390 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=8 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 8-193273528390 2024-09-17T19:18:01.413 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528390 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.8 seq 193273528390' 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.8 seq 193273528390 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 8 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528391 -lt 193273528390 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:18:01.414 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 9-450971566096 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=9 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 9-450971566096 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566096 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.9 seq 450971566096' 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: waiting osd.9 seq 450971566096 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 9 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566097 -lt 450971566096 2024-09-17T19:18:01.415 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 5 == 0 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:18:01.416 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:18:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:18:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:18:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:18:02.085 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:254139: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 5 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:254139: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: run_in_background: return 0 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: for pid in $pids 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2175: wait_background: wait 254138 2024-09-17T19:18:02.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: eval 'pids='\'''\''' 2024-09-17T19:18:02.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: wait_background: pids= 2024-09-17T19:18:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: wait_background: return 0 2024-09-17T19:18:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-09-17T19:18:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-09-17T19:18:02.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:18:02.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:18:02.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T19:18:02.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:18:02.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:18:02.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:18:02.162 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:18:02.162 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:18:02.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:18:02.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:18:02.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:18:02.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:18:02.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:18:02.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:18:02.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:18:02.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:18:02.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:18:02.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:18:02.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:18:02.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:18:02.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:18:02.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:18:02.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:18:02.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:18:02.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:18:02.353 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:18:02.353 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:02.353 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:02.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:18:02.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:18:02.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:18:02.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:18:02.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:18:02.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:18:02.356 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:18:02.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:18:02.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:18:02.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:18:02.358 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:18:02.359 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:18:02.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:18:02.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:18:02.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:18:02.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:18:02.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:18:02.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:18:02.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:18:02.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:18:02.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:02.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:02.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:18:02.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:18:02.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:18:02.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T19:18:02.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T19:18:02.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:02.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:02.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T19:18:02.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T19:18:02.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-17T19:18:02.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T19:18:02.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:18:02.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:18:02.845 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:18:02.845 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:02.845 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:02.845 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:18:02.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-17T19:18:03.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:18:03.050 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:18:03.051 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:18:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:03.052 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:18:03.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:18:03.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:18:03.166 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:03.167 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:03.167 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:18:03.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:18:03.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T19:18:03.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:04.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:18:04.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:18:04.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:18:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-09-17T19:18:04.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:18:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:18:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:18:04.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:18:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=302c24d2-6f6f-496e-85d7-417d35da43c5 2024-09-17T19:18:04.190 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 302c24d2-6f6f-496e-85d7-417d35da43c5 2024-09-17T19:18:04.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 302c24d2-6f6f-496e-85d7-417d35da43c5' 2024-09-17T19:18:04.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:18:04.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDs1elmbydiDBAAuEWFI0tYh2yhyd2B77F3yw== 2024-09-17T19:18:04.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDs1elmbydiDBAAuEWFI0tYh2yhyd2B77F3yw=="}' 2024-09-17T19:18:04.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 302c24d2-6f6f-496e-85d7-417d35da43c5 -i td/osd-scrub-repair/0/new.json 2024-09-17T19:18:04.441 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:18:04.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T19:18:04.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDs1elmbydiDBAAuEWFI0tYh2yhyd2B77F3yw== --osd-uuid 302c24d2-6f6f-496e-85d7-417d35da43c5 2024-09-17T19:18:04.487 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:04.485+0000 7f03478257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:04.491 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:04.493+0000 7f03478257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:04.493 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:04.493+0000 7f03478257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:04.493 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:04.493+0000 7f03478257c0 -1 bdev(0x55d8b40cdc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T19:18:04.493 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:04.493+0000 7f03478257c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T19:18:10.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T19:18:10.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:18:10.702 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T19:18:10.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T19:18:10.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:18:11.125 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T19:18:11.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T19:18:11.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:18:11.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:18:11.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:18:11.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:18:11.158 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:11.157+0000 7fa8c593d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:11.160 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:11.161+0000 7fa8c593d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:11.161 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:11.161+0000 7fa8c593d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:11.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T19:18:11.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:18:11.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:18:11.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:18:11.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:18:11.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:11.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:18:11.474 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:18:11.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:11.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:11.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:12.642 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:12.641+0000 7fa8c593d7c0 -1 Falling back to public interface 2024-09-17T19:18:12.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:12.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:12.807 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:18:12.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:18:12.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:12.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:13.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:14.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:14.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:14.124 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:18:14.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:18:14.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:14.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:14.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:15.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:15.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:15.449 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:18:15.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:18:15.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:15.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:15.597 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:15.597+0000 7fa8c593d7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T19:18:15.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:16.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:16.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:16.832 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:18:16.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:18:16.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:16.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:17.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:18.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:18.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:18.166 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:18:18.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:18:18.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:18.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:18.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:19.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:19.494 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:18:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:18:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:19.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:19.814 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:20.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:20.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:20.817 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:18:20.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:18:20.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:20.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:21.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:22.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:22.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:22.143 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:18:22.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:18:22.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:22.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:22.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:23.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:23.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:23.509 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:18:23.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:18:23.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:23.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:23.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:24.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:24.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:24.833 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:18:24.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:18:24.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:24.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:25.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:26.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:26.155 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:18:26.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:18:26.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:26.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:26.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:27.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:27.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:27.482 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:18:27.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:18:27.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:27.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:28.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:29.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:29.090 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:18:29.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:18:29.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:29.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:29.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:30.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:30.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:30.959 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:18:30.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:18:30.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:30.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:31.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:32.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:32.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:32.436 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:18:32.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:18:32.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:32.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:33.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:34.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:34.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:34.011 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:18:34.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:18:34.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:34.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:34.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:35.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:35.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:35.602 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:18:35.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:18:35.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:35.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:36.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:37.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:37.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:37.085 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:18:37.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:18:37.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:37.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:37.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:38.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:38.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:38.931 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:18:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:18:38.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:18:39.291 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/158884328,v1:127.0.0.1:6803/158884328] [v2:127.0.0.1:6804/158884328,v1:127.0.0.1:6805/158884328] exists,up 302c24d2-6f6f-496e-85d7-417d35da43c5 2024-09-17T19:18:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:18:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:18:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:18:39.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:18:39.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:18:39.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:18:39.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:18:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:18:39.298 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:18:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:18:39.298 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:18:39.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7572f8c5-fbc9-4863-816a-b919c97a75bc' 2024-09-17T19:18:39.298 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:18:39.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAP1ulmH2cpExAAsuj8PVxgGBbrUg3YYF+ELQ== 2024-09-17T19:18:39.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAP1ulmH2cpExAAsuj8PVxgGBbrUg3YYF+ELQ=="}' 2024-09-17T19:18:39.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7572f8c5-fbc9-4863-816a-b919c97a75bc -i td/osd-scrub-repair/1/new.json 2024-09-17T19:18:39.726 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:18:39.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T19:18:39.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAP1ulmH2cpExAAsuj8PVxgGBbrUg3YYF+ELQ== --osd-uuid 7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:18:39.782 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:39.781+0000 7fa06c08e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:39.827 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:39.829+0000 7fa06c08e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:39.829 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:39.829+0000 7fa06c08e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:39.830 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:39.829+0000 7fa06c08e7c0 -1 bdev(0x55e946fadc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T19:18:39.830 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:39.829+0000 7fa06c08e7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T19:18:46.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T19:18:46.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:18:46.446 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T19:18:46.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T19:18:46.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:18:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T19:18:46.845 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T19:18:46.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:18:46.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:18:46.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:18:46.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:18:46.877 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:46.876+0000 7ff0d0e5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:46.879 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:46.880+0000 7ff0d0e5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:46.880 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:46.880+0000 7ff0d0e5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:18:47.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T19:18:47.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:18:47.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:18:47.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:18:47.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:18:47.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:47.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:18:47.196 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:18:47.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:47.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:47.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:47.902 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:47.900+0000 7ff0d0e5d7c0 -1 Falling back to public interface 2024-09-17T19:18:48.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:48.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:48.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:18:48.538 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:18:48.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:48.538 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:48.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:49.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:49.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:49.871 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:18:49.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:18:49.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:49.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:50.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:50.704 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:18:50.704+0000 7ff0d0e5d7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T19:18:51.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:51.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:51.212 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:18:51.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:18:51.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:51.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:51.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:52.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:52.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:52.721 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:18:52.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:18:52.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:52.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:53.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:54.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:54.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:54.046 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:18:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:18:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:54.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:54.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:55.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:55.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:55.379 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:18:55.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:18:55.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:55.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:55.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:56.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:56.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:56.803 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:18:56.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:18:56.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:56.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:57.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:58.219 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:58.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:58.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:18:58.220 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:18:58.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:58.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:58.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:18:59.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:18:59.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:18:59.561 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:18:59.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:18:59.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:18:59.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:18:59.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:00.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:00.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:00.894 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:19:00.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:19:00.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:00.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:01.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:02.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:02.216 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:02.216 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:19:02.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:19:02.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:02.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:02.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:03.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:03.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:03.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:19:03.568 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:19:03.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:03.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:04.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:05.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:05.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:05.072 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:19:05.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:19:05.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:05.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:06.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:07.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:07.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:07.159 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:19:07.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:19:07.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:07.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:07.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:08.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:08.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:08.904 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:19:08.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:19:08.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:08.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:09.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:10.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:10.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:10.386 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:19:10.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:19:10.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:10.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:10.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:11.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:11.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:11.970 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:19:11.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:19:11.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:11.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:12.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:13.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:13.671 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:13.672 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:19:13.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:19:13.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:13.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:14.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:15.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:15.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:15.035 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:19:15.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:19:15.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:15.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:15.372 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1617901066,v1:127.0.0.1:6811/1617901066] [v2:127.0.0.1:6812/1617901066,v1:127.0.0.1:6813/1617901066] exists,up 7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:19:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:19:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:19:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:19:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-09-17T19:19:15.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T19:19:15.691 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T19:19:15.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T19:19:15.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T19:19:16.170 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T19:19:16.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:19:17.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T19:19:19.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-09-17T19:19:19.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:19:19.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:19:19.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:19:19.351 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:19:19.351 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:19:19.351 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:19:19.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:19:19.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:19:19.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:19:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:19:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:19:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:19:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:19:19.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:19:19.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:19:19.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:19:19.813 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:19:19.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:19:19.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:19:19.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:19:19.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-17T19:19:19.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-17T19:19:19.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-17T19:19:19.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:19:19.971 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:19:20.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672966 2024-09-17T19:19:20.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672966 2024-09-17T19:19:20.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492 1-42949672966' 2024-09-17T19:19:20.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:19:20.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:19:20.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-17T19:19:20.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:19:20.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-17T19:19:20.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:19:20.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-17T19:19:20.115 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836492 2024-09-17T19:19:20.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-17T19:19:20.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:19:20.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-17T19:19:20.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:19:21.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:19:21.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:19:21.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-17T19:19:21.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:19:21.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672966 2024-09-17T19:19:21.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:19:21.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:19:21.771 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672966 2024-09-17T19:19:21.771 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:19:21.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672966 2024-09-17T19:19:21.773 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672966 2024-09-17T19:19:21.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-09-17T19:19:21.773 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:19:22.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672966 2024-09-17T19:19:22.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:19:22.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:19:22.123 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:19:22.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:19:22.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:19:22.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:19:22.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:19:22.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:19:22.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:19:22.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:19:22.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:19:22.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:19:22.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:19:22.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:19:22.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:19:23.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:19:23.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:19:23.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:19:23.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:19:23.689 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:19:23.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:19:24.309 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:19:24.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:19:24.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:19:24.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-09-17T19:19:24.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-09-17T19:19:24.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2024-09-17T19:19:24.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2024-09-17T19:19:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2024-09-17T19:19:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-17T19:19:24.710 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T19:19:24.711 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-17T19:19:24.711 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T19:19:25.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-17T19:19:25.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-09-17T19:19:25.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-17T19:19:25.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-09-17T19:19:25.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T19:19:25.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-17T19:19:25.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-09-17T19:19:25.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T19:19:25.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:19:25.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:19:25.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:19:25.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-09-17T19:19:27.508 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-17T19:19:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:19:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:19:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:19:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:19:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:19:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:19:28.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:19:28.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:19:28.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:19:28.047 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:19:28.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:19:28.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:19:28.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:19:28.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:19:28.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:19:28.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:19:28.087 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:28.088+0000 7f317e9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:28.089 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:28.088+0000 7f317e9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:28.092+0000 7f317e9ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:28.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:19:28.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:19:28.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:19:28.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:19:28.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:19:28.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:28.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:19:28.361 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:19:28.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:19:28.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:28.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:29.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:29.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:29.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:19:29.702 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:19:29.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:29.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:19:29.848 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:29.848+0000 7f317e9ff7c0 -1 Falling back to public interface 2024-09-17T19:19:30.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:31.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:31.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:31.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:19:31.046 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:19:31.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:31.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:19:31.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:32.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:32.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:32.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:19:32.369 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:19:32.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:32.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:19:32.500 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:32.500+0000 7f317e9ff7c0 -1 osd.0 22 log_to_monitors true 2024-09-17T19:19:32.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:33.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:33.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:33.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:19:33.724 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:19:33.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:33.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2654984410,v1:127.0.0.1:6803/2654984410] [v2:127.0.0.1:6804/2654984410,v1:127.0.0.1:6805/2654984410] exists,up 302c24d2-6f6f-496e-85d7-417d35da43c5 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:19:34.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:19:34.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:19:34.092 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:19:34.092 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:19:34.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:19:34.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:19:34.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:19:34.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:19:34.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:19:34.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:19:34.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:19:34.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:19:34.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:19:34.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:19:34.513 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:19:34.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:19:34.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:19:34.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:19:34.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149698 2024-09-17T19:19:34.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149698 2024-09-17T19:19:34.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698' 2024-09-17T19:19:34.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:19:34.681 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:19:34.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672970 2024-09-17T19:19:34.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672970 2024-09-17T19:19:34.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149698 1-42949672970' 2024-09-17T19:19:34.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:19:34.821 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149698 2024-09-17T19:19:34.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:19:34.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:19:34.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149698 2024-09-17T19:19:34.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:19:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149698 2024-09-17T19:19:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-09-17T19:19:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 111669149698 2024-09-17T19:19:34.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:19:35.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149698 -lt 111669149698 2024-09-17T19:19:35.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:19:35.162 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672970 2024-09-17T19:19:35.162 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:19:35.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:19:35.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672970 2024-09-17T19:19:35.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:19:35.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672970 2024-09-17T19:19:35.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-09-17T19:19:35.166 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 42949672970 2024-09-17T19:19:35.166 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:19:35.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672970 -lt 42949672970 2024-09-17T19:19:35.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:19:35.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:19:35.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:19:35.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:19:35.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:19:36.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:19:36.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:19:36.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:19:36.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:19:36.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:19:36.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:19:36.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:19:36.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-17T19:19:36.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-17T19:19:36.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:19:36.573 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:19:36.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-17T19:19:36.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-17T19:19:36.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-17T19:19:36.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-17T19:19:36.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-17T19:19:36.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:19:36.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:19:36.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:19:36.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:19:37.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:37.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-17T19:19:37.590 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-17T19:19:37.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:37.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-17T19:19:37.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:37.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:19:37.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:19:37.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:19:37.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:19:37.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:19:37.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:19:37.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:19:37.608 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:19:37.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:16.090348+0000 '>' 2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:37.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:19:38.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:19:38.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:19:38.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:19:38.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:19:38.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:19:38.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:19:38.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:19:39.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:16.090348+0000 '>' 2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:39.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:19:40.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:19:40.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:19:40.249 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:19:40.249 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:19:40.249 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:19:40.250 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:19:40.250 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:19:40.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:16.090348+0000 '>' 2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:40.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:19:41.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:19:41.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:19:41.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:19:41.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:19:41.585 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:19:41.585 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:19:41.585 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:19:41.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:38.319987+0000 '>' 2024-09-17T19:19:16.090348+0000 2024-09-17T19:19:41.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:19:41.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:19:43.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:19:43.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:19:43.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:19:43.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:19:43.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:19:43.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:19:43.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:19:44.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:19:44.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T19:19:44.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:19:44.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:19:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:19:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:19:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:19:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T19:19:44.744 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T19:19:44.744 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T19:19:44.744 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:19:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:19:45.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:19:45.032 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:19:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:19:45.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:19:45.034 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:19:45.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:19:45.034 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:19:45.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:19:45.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:19:45.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:19:45.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:19:45.068 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:45.068+0000 7feb9d76d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:45.071 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:45.072+0000 7feb9d76d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:45.073 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:45.072+0000 7feb9d76d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:45.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:45.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:46.325 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:46.323+0000 7feb9d76d7c0 -1 Falling back to public interface 2024-09-17T19:19:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:19:46.739 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:19:46.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:46.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:47.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:48.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:48.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:48.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:19:48.068 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:19:48.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:48.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:48.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:49.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:49.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:49.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:19:49.406 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:19:49.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:49.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:49.578 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:49.575+0000 7feb9d76d7c0 -1 osd.1 28 log_to_monitors true 2024-09-17T19:19:49.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:50.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:50.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:50.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:19:50.832 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:19:50.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:50.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:51.242 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3767613094,v1:127.0.0.1:6811/3767613094] [v2:127.0.0.1:6812/3767613094,v1:127.0.0.1:6813/3767613094] exists,up 7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:19:51.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:19:51.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:19:51.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:19:51.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:19:51.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:19:51.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:19:51.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:19:51.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:19:51.244 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:19:51.244 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:19:51.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:19:51.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:19:51.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:19:51.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:19:51.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:19:51.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:19:51.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:19:51.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:19:51.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:19:51.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:19:51.662 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:19:51.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:19:51.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:19:51.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:19:51.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149702 2024-09-17T19:19:51.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149702 2024-09-17T19:19:51.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702' 2024-09-17T19:19:51.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:19:51.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:19:51.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953474 2024-09-17T19:19:51.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953474 2024-09-17T19:19:51.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-09-17T19:19:51.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:19:51.970 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:19:51.970 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149702 2024-09-17T19:19:51.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:19:51.972 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149702 2024-09-17T19:19:51.972 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:19:51.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149702 2024-09-17T19:19:51.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-09-17T19:19:51.974 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 111669149702 2024-09-17T19:19:51.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:19:52.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149701 -lt 111669149702 2024-09-17T19:19:52.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:19:53.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:19:53.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:19:53.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149703 -lt 111669149702 2024-09-17T19:19:53.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:19:53.628 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:19:53.628 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953474 2024-09-17T19:19:53.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:19:53.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953474 2024-09-17T19:19:53.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:19:53.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953474 2024-09-17T19:19:53.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-09-17T19:19:53.632 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 137438953474 2024-09-17T19:19:53.632 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:19:53.952 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953474 -lt 137438953474 2024-09-17T19:19:53.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:19:53.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:19:53.954 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:19:54.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:19:54.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:19:54.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:19:54.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:19:54.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:19:54.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:19:54.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:19:54.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:19:54.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:19:54.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:19:54.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:19:54.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:19:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:19:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:19:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:19:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:19:55.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:19:55.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-09-17T19:19:55.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2024-09-17T19:19:55.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2024-09-17T19:19:55.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2024-09-17T19:19:55.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:19:55.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:19:55.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:19:55.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:19:55.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-09-17T19:19:57.522 INFO:tasks.workunit.client.0.smithi105.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-09-17T19:19:58.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:19:58.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:19:58.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:19:58.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:19:58.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:19:58.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:19:58.059 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:19:58.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:19:58.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:19:58.060 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:19:58.060 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:19:58.060 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:19:58.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:19:58.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:19:58.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:19:58.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:19:58.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:19:58.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:19:58.095 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:58.095+0000 7f2887e9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:58.098 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:58.095+0000 7f2887e9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:58.099 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:58.099+0000 7f2887e9e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:19:58.394 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:19:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:19:58.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:19:59.414 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:19:59.411+0000 7f2887e9e7c0 -1 Falling back to public interface 2024-09-17T19:19:59.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:19:59.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:19:59.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:19:59.731 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:19:59.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:19:59.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:00.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:01.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:01.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:01.056 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:20:01.056 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:20:01.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:01.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:01.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:02.093 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:02.091+0000 7f2887e9e7c0 -1 osd.1 34 log_to_monitors true 2024-09-17T19:20:02.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:02.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:02.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:20:02.404 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:20:02.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:02.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 38 up_thru 32 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/2840174679,v1:127.0.0.1:6811/2840174679] [v2:127.0.0.1:6812/2840174679,v1:127.0.0.1:6813/2840174679] exists,up 7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:20:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:20:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:20:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:20:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:20:02.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:20:02.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:20:02.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:20:02.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:20:02.858 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:20:02.858 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:20:02.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:20:02.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:20:02.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:20:02.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:20:02.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:20:02.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:20:02.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:20:02.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:20:02.972 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:20:03.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:20:03.405 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:20:03.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:20:03.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:20:03.405 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:20:03.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149706 2024-09-17T19:20:03.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149706 2024-09-17T19:20:03.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149706' 2024-09-17T19:20:03.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:20:03.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:20:03.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757250 2024-09-17T19:20:03.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757250 2024-09-17T19:20:03.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149706 1-163208757250' 2024-09-17T19:20:03.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:20:03.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149706 2024-09-17T19:20:03.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:20:03.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:20:03.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149706 2024-09-17T19:20:03.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:20:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149706 2024-09-17T19:20:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149706' 2024-09-17T19:20:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 111669149706 2024-09-17T19:20:03.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:20:04.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149705 -lt 111669149706 2024-09-17T19:20:04.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:20:05.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:20:05.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:20:05.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149706 -lt 111669149706 2024-09-17T19:20:05.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:20:05.366 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-163208757250 2024-09-17T19:20:05.366 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:20:05.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:20:05.368 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-163208757250 2024-09-17T19:20:05.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:20:05.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757250 2024-09-17T19:20:05.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-09-17T19:20:05.370 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 163208757250 2024-09-17T19:20:05.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:20:05.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757250 -lt 163208757250 2024-09-17T19:20:05.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:20:05.701 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:20:05.701 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:20:06.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:20:06.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:20:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:20:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:20:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:20:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:20:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:20:06.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:20:06.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:20:06.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:20:06.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:20:06.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:20:06.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:20:06.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:20:06.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:20:06.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-09-17T19:20:06.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2024-09-17T19:20:06.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2024-09-17T19:20:06.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2024-09-17T19:20:06.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:20:07.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-09-17T19:20:07.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-09-17T19:20:07.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=1.3 2024-09-17T19:20:07.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 1.3 2024-09-17T19:20:07.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:20:07.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:20:07.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:20:07.173 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:20:07.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:07.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 1.3 2024-09-17T19:20:07.782 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 1.3 on osd.1 to repair 2024-09-17T19:20:07.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 1.3 2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:07.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=1.3 2024-09-17T19:20:07.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:07.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:20:07.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:20:08.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:38.319987+0000 '>' 2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:08.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:20:09.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:20:09.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:20:09.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:20:09.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:20:09.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:20:09.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:20:09.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:20:09.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:38.319987+0000 '>' 2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:09.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:20:10.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:20:10.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:20:10.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:20:10.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:20:10.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:20:10.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:20:10.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:20:10.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:38.319987+0000 '>' 2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:10.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:20:11.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:20:11.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:20:11.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:20:11.756 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:20:11.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:20:11.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:20:11.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:20:12.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:19:38.319987+0000 '>' 2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:12.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:20:13.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:20:13.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:20:13.090 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-09-17T19:20:13.090 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2024-09-17T19:20:13.090 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:20:13.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:20:13.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-09-17T19:20:13.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:20:08.157546+0000 '>' 2024-09-17T19:19:38.319987+0000 2024-09-17T19:20:13.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:20:13.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:20:15.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:20:15.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:20:15.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:20:15.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:20:15.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:20:15.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:20:15.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:20:15.724 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-09-17T19:20:16.223 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-09-17T19:20:16.223 INFO:tasks.workunit.client.0.smithi105.stdout:_ 2024-09-17T19:20:16.223 INFO:tasks.workunit.client.0.smithi105.stdout:snapset 2024-09-17T19:20:16.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:20:16.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:16.515 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:20:16.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:20:16.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:20:16.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:20:16.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:20:16.518 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:20:16.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:20:16.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:20:16.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:20:16.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:20:16.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:20:16.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:20:16.551 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:16.551+0000 7f22d53277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:16.554 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:16.551+0000 7f22d53277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:16.556 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:16.555+0000 7f22d53277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:16.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:20:16.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:20:16.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:20:16.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:20:16.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:20:16.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:16.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:20:16.847 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:20:16.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:16.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:17.606 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:17.603+0000 7f22d53277c0 -1 Falling back to public interface 2024-09-17T19:20:18.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:18.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:18.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:20:18.223 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:20:18.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:18.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:18.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:19.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:19.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:19.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:20:19.546 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:20:19.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:19.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:19.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:20.844 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:20.843+0000 7f22d53277c0 -1 osd.1 39 log_to_monitors true 2024-09-17T19:20:20.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:20.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:20.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:20:20.890 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:20:20.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:20.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:21.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:22.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:22.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:22.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:20:22.226 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:20:22.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:22.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:20:22.608 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/2092866947,v1:127.0.0.1:6811/2092866947] [v2:127.0.0.1:6812/2092866947,v1:127.0.0.1:6813/2092866947] exists,up 7572f8c5-fbc9-4863-816a-b919c97a75bc 2024-09-17T19:20:22.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:20:22.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:20:22.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:20:22.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:20:22.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:20:22.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:20:22.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:20:22.610 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:20:22.610 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:20:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:20:22.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:20:22.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:20:22.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:20:22.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:20:22.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:20:22.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:20:22.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:20:22.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:20:22.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:20:23.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:20:23.030 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:20:23.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:20:23.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:20:23.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:20:23.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=111669149711 2024-09-17T19:20:23.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 111669149711 2024-09-17T19:20:23.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149711' 2024-09-17T19:20:23.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:20:23.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:20:23.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=184683593730 2024-09-17T19:20:23.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 184683593730 2024-09-17T19:20:23.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-111669149711 1-184683593730' 2024-09-17T19:20:23.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:20:23.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:20:23.374 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-111669149711 2024-09-17T19:20:23.375 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:20:23.375 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-111669149711 2024-09-17T19:20:23.376 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:20:23.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=111669149711 2024-09-17T19:20:23.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 111669149711' 2024-09-17T19:20:23.377 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 111669149711 2024-09-17T19:20:23.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:20:23.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149710 -lt 111669149711 2024-09-17T19:20:23.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:20:24.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:20:24.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:20:25.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 111669149711 -lt 111669149711 2024-09-17T19:20:25.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:20:25.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-184683593730 2024-09-17T19:20:25.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:20:25.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:20:25.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-184683593730 2024-09-17T19:20:25.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:20:25.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=184683593730 2024-09-17T19:20:25.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-09-17T19:20:25.014 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 184683593730 2024-09-17T19:20:25.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:20:25.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 184683593730 -lt 184683593730 2024-09-17T19:20:25.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:20:25.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:20:25.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:20:25.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:20:25.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:20:25.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:20:25.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:20:25.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:20:25.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:20:25.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:20:25.732 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:20:26.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:20:26.074 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:20:26.074 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:20:26.074 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:20:26.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:20:26.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:20:26.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:20:26.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-09-17T19:20:26.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:20:26.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:20:26.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:20:26.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:20:26.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:20:26.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:20:26.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:20:26.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:20:26.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:20:26.625 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:20:26.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:20:26.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:20:26.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:20:26.626 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:20:26.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:20:26.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:20:26.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:20:26.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:20:26.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:26.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:26.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:20:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:20:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:20:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T19:20:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T19:20:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T19:20:26.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:20:26.644 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:20:26.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:20:26.645 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:20:26.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:20:26.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:20:26.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:20:26.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:20:26.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:20:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:20:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:20:26.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:20:26.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:20:26.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:20:26.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:20:26.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:20:26.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:20:26.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:26.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:26.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:20:26.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:20:26.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:20:26.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T19:20:26.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T19:20:26.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:26.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:26.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T19:20:26.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T19:20:26.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T19:20:26.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T19:20:26.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T19:20:26.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T19:20:27.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T19:20:27.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:20:27.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:20:27.136 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:20:27.136 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:27.136 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:27.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:20:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T19:20:27.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T19:20:27.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T19:20:27.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:20:27.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:20:27.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T19:20:27.351 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T19:20:27.351 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:20:27.351 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:20:27.351 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:20:27.352 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:20:27.352 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:27.352 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:27.352 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:20:27.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:20:27.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T19:20:27.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T19:20:27.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:20:27.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:20:27.469 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T19:20:27.470 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:20:27.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T19:20:27.470 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:20:27.470 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:20:27.471 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:20:27.471 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:27.471 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:27.471 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:20:27.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:20:27.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T19:20:27.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-17T19:20:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T19:20:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T19:20:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T19:20:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T19:20:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T19:20:27.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T19:20:28.448 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T19:20:28.448 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:20:28.448 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:20:28.449 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:20:28.449 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:28.449 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:28.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:20:28.449 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:20:28.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:20:28.487 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-17T19:20:28.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-17T19:20:28.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-17T19:20:28.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:20:28.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:20:28.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:20:28.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:20:28.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7e026281-87cb-482c-9e9d-db7121e47cfe' 2024-09-17T19:20:28.494 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:20:28.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:20:28.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB81ulmbvadHhAAI8wEpFFB/DA1h2w7lw9YxQ== 2024-09-17T19:20:28.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB81ulmbvadHhAAI8wEpFFB/DA1h2w7lw9YxQ=="}' 2024-09-17T19:20:28.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e026281-87cb-482c-9e9d-db7121e47cfe -i td/osd-scrub-repair/0/new.json 2024-09-17T19:20:28.752 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:20:28.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T19:20:28.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB81ulmbvadHhAAI8wEpFFB/DA1h2w7lw9YxQ== --osd-uuid 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:20:28.799 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:28.799+0000 7fe6f6aed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:28.802 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:28.799+0000 7fe6f6aed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:28.803 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:28.803+0000 7fe6f6aed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:28.803 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:28.803+0000 7fe6f6aed7c0 -1 bdev(0x560f8fc31c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T19:20:28.803 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:28.803+0000 7fe6f6aed7c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T19:20:35.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T19:20:35.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:20:35.134 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T19:20:35.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T19:20:35.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:20:35.551 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T19:20:35.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T19:20:35.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:20:35.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:20:35.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:20:35.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:20:35.586 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:35.582+0000 7f04ce21a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:35.608 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:35.606+0000 7f04ce21a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:35.609 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:35.606+0000 7f04ce21a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:20:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T19:20:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:20:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:20:35.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:20:35.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:20:35.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:35.876 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:20:35.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:20:35.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:35.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:36.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:36.641 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:36.638+0000 7f04ce21a7c0 -1 Falling back to public interface 2024-09-17T19:20:37.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:37.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:37.201 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:20:37.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:20:37.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:37.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:37.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:38.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:38.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:38.536 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:20:38.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:20:38.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:38.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:38.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:39.372 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:20:39.370+0000 7f04ce21a7c0 -1 osd.0 0 log_to_monitors true 2024-09-17T19:20:39.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:39.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:39.865 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:20:39.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:20:39.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:39.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:40.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:41.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:41.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:41.450 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:20:41.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:20:41.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:41.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:41.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:42.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:42.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:42.767 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:20:42.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:20:42.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:42.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:43.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:44.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:44.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:44.205 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:20:44.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:20:44.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:44.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:44.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:45.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:45.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:45.544 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:20:45.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:20:45.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:45.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:45.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:46.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:46.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:46.874 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:20:46.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:20:46.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:46.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:47.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:48.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:48.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:48.210 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:20:48.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:20:48.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:48.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:48.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:49.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:49.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:49.544 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:20:49.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:20:49.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:49.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:49.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:50.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:50.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:50.872 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:20:50.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:20:50.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:50.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:51.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:52.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:52.197 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:52.197 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:20:52.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:20:52.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:52.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:52.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:53.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:53.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:53.793 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:20:53.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:20:53.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:53.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:54.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:55.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:55.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:55.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:20:55.418 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:20:55.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:55.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:56.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:57.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:57.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:57.394 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:20:57.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:20:57.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:57.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:20:58.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:20:59.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:20:59.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:20:59.281 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:20:59.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:20:59.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:20:59.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:21:00.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:01.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:01.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:01.109 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:21:01.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:21:01.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:01.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:21:01.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:02.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:02.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:02.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:21:02.918 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:21:02.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:02.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:21:03.247 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2353639553,v1:127.0.0.1:6803/2353639553] [v2:127.0.0.1:6804/2353639553,v1:127.0.0.1:6805/2353639553] exists,up 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:21:03.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:21:03.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:21:03.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:21:03.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-17T19:21:03.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:21:03.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:21:03.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:21:03.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:21:03.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:21:03.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:21:03.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1052c18e-53c6-42d0-9f5f-81716a682fc7' 2024-09-17T19:21:03.255 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:21:03.256 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:21:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCf1ulmJ4U/EBAA1p8zKLttdKBoitPjGIoJ0Q== 2024-09-17T19:21:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCf1ulmJ4U/EBAA1p8zKLttdKBoitPjGIoJ0Q=="}' 2024-09-17T19:21:03.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1052c18e-53c6-42d0-9f5f-81716a682fc7 -i td/osd-scrub-repair/1/new.json 2024-09-17T19:21:03.718 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:21:03.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T19:21:03.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCf1ulmJ4U/EBAA1p8zKLttdKBoitPjGIoJ0Q== --osd-uuid 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:21:03.776 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:03.774+0000 7f85155967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:03.811 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:03.810+0000 7f85155967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:03.813 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:03.810+0000 7f85155967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:03.813 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:03.810+0000 7f85155967c0 -1 bdev(0x557392c0dc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T19:21:03.813 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:03.810+0000 7f85155967c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T19:21:10.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T19:21:10.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:21:10.042 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T19:21:10.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T19:21:10.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:21:10.440 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T19:21:10.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T19:21:10.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:21:10.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:21:10.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:21:10.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:21:10.473 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:10.474+0000 7fc5955017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:10.506 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:10.506+0000 7fc5955017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:10.508 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:10.510+0000 7fc5955017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:10.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:21:10.783 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:21:10.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:10.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:11.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:12.034 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:12.034+0000 7fc5955017c0 -1 Falling back to public interface 2024-09-17T19:21:12.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:12.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:12.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:21:12.125 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:21:12.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:12.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:12.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:13.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:13.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:13.447 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:21:13.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:21:13.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:13.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:13.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:14.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:14.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:14.782 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:21:14.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:21:14.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:14.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:14.949 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:14.949+0000 7fc5955017c0 -1 osd.1 0 log_to_monitors true 2024-09-17T19:21:15.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:16.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:16.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:16.187 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:21:16.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:21:16.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:16.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:16.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:17.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:17.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:17.524 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:21:17.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:21:17.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:17.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:17.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:18.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:18.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:18.849 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:21:18.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:21:18.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:18.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:19.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:20.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:20.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:20.185 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:21:20.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:21:20.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:20.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:20.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:21.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:21.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:21.518 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:21:21.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:21:21.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:21.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:21.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:22.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:22.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:22.969 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:21:22.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:21:22.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:22.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:23.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:24.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:24.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:24.298 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:21:24.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:21:24.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:24.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:24.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:25.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:25.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:25.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:21:25.638 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:21:25.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:25.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:26.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:27.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:27.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:27.026 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:21:27.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:21:27.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:27.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:28.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:28.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:28.365 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:21:28.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:21:28.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:28.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:28.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:30.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:30.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:30.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:21:30.001 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:21:30.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:30.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:30.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:31.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:31.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:31.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:21:31.579 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:21:31.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:31.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:31.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:32.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:32.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:32.986 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:21:32.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:21:32.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:32.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:33.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:34.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:34.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:34.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:21:34.693 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:21:34.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:34.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:35.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:36.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:36.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:36.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:21:36.489 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:21:36.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:36.489 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:37.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:38.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:38.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:38.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:21:38.156 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:21:38.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:38.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:38.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:39.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:39.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:39.482 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:21:39.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:21:39.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:39.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:39.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:40.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:40.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:40.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T19:21:40.959 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T19:21:40.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:40.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:21:41.301 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4285109958,v1:127.0.0.1:6811/4285109958] [v2:127.0.0.1:6812/4285109958,v1:127.0.0.1:6813/4285109958] exists,up 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:21:41.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:21:41.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:21:41.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:21:41.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:21:41.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:21:41.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:21:41.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:21:41.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6b05f872-817c-4060-aa7f-3ccd7598ee4e 2024-09-17T19:21:41.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6b05f872-817c-4060-aa7f-3ccd7598ee4e' 2024-09-17T19:21:41.308 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 6b05f872-817c-4060-aa7f-3ccd7598ee4e 2024-09-17T19:21:41.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:21:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDF1ulmkQ28ExAAMugo0RG3gEGQrKwbtScalg== 2024-09-17T19:21:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDF1ulmkQ28ExAAMugo0RG3gEGQrKwbtScalg=="}' 2024-09-17T19:21:41.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6b05f872-817c-4060-aa7f-3ccd7598ee4e -i td/osd-scrub-repair/2/new.json 2024-09-17T19:21:41.715 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:21:41.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T19:21:41.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDF1ulmkQ28ExAAMugo0RG3gEGQrKwbtScalg== --osd-uuid 6b05f872-817c-4060-aa7f-3ccd7598ee4e 2024-09-17T19:21:41.769 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:41.769+0000 7fbed6bc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:41.799 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:41.801+0000 7fbed6bc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:41.801 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:41.801+0000 7fbed6bc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:41.802 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:41.801+0000 7fbed6bc67c0 -1 bdev(0x55689acc9c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T19:21:41.802 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:41.801+0000 7fbed6bc67c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T19:21:48.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T19:21:48.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:21:48.617 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T19:21:48.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T19:21:48.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:21:49.018 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T19:21:49.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T19:21:49.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:21:49.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:21:49.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:21:49.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:21:49.050 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:49.049+0000 7fa6946977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:49.052 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:49.053+0000 7fa6946977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:49.053 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:49.053+0000 7fa6946977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:21:49.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T19:21:49.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:21:49.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:21:49.359 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:21:49.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:21:49.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:21:49.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:49.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:21:49.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:49.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:49.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:50.314 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:50.313+0000 7fa6946977c0 -1 Falling back to public interface 2024-09-17T19:21:50.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:50.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:50.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:21:50.695 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:21:50.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:50.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:51.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:52.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:52.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:52.026 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:21:52.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:21:52.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:52.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:52.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:53.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:53.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:53.355 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:21:53.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:21:53.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:53.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:53.370 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:21:53.369+0000 7fa6946977c0 -1 osd.2 0 log_to_monitors true 2024-09-17T19:21:53.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:54.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:54.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:54.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:21:54.709 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:21:54.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:54.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:55.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:56.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:56.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:56.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:21:56.046 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:21:56.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:56.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:56.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:57.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:57.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:57.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:21:57.382 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:21:57.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:57.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:57.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:21:58.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:21:58.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:21:58.847 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:21:58.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:21:58.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:21:58.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:21:59.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:00.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:00.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:00.177 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:22:00.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:22:00.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:00.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:00.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:01.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:01.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:01.500 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:22:01.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:22:01.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:01.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:01.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:02.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:02.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:02.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:22:02.876 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:22:02.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:02.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:03.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:04.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:04.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:04.213 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:22:04.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:22:04.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:04.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:04.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:05.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:05.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:22:05.552 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:22:05.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:05.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:05.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:06.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:06.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:06.878 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:22:06.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:22:06.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:06.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:07.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:08.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:08.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:08.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:22:08.195 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:22:08.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:08.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:08.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:09.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:09.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:09.829 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:22:09.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:22:09.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:09.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:10.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:11.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:11.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:11.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:22:11.637 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:22:11.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:11.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:12.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:13.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:13.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:13.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:22:13.480 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:22:13.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:13.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:14.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:15.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:15.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:15.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:22:15.277 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:22:15.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:15.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:15.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:16.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:16.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:16.897 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:22:16.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:22:16.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:16.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:17.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:18.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:18.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:18.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:22:18.407 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:22:18.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:18.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:18.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:19.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:19.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:19.744 INFO:tasks.workunit.client.0.smithi105.stdout:21 2024-09-17T19:22:19.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 21 2024-09-17T19:22:19.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:19.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:22:20.088 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/755441863,v1:127.0.0.1:6819/755441863] [v2:127.0.0.1:6820/755441863,v1:127.0.0.1:6821/755441863] exists,up 6b05f872-817c-4060-aa7f-3ccd7598ee4e 2024-09-17T19:22:20.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:22:20.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:22:20.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:22:20.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-17T19:22:20.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T19:22:20.413 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T19:22:20.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T19:22:20.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T19:22:20.850 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T19:22:20.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:22:21.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T19:22:24.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-17T19:22:24.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-17T19:22:24.693 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'foo' created 2024-09-17T19:22:24.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:22:25.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-09-17T19:22:25.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T19:22:25.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T19:22:25.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=false 2024-09-17T19:22:25.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T19:22:25.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-17T19:22:26.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T19:22:26.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T19:22:26.995 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T19:22:27.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:22:28.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' false = true ']' 2024-09-17T19:22:28.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T19:22:28.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:22:28.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:22:28.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:22:28.021 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:22:28.022 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:22:28.022 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:22:28.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:22:28.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:22:28.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:22:28.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:22:28.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:22:28.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:22:28.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:22:28.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:22:28.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:22:28.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:22:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:22:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:22:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:22:28.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:28.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:22:28.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-17T19:22:28.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-17T19:22:28.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-17T19:22:28.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:28.708 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:22:28.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672980 2024-09-17T19:22:28.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672980 2024-09-17T19:22:28.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672980' 2024-09-17T19:22:28.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:28.907 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:22:29.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-17T19:22:29.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-17T19:22:29.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672980 2-64424509449' 2024-09-17T19:22:29.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:29.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-17T19:22:29.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:29.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:22:29.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-17T19:22:29.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:29.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-17T19:22:29.083 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836505 2024-09-17T19:22:29.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-17T19:22:29.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:22:29.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-17T19:22:29.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:29.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:29.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672980 2024-09-17T19:22:29.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:22:29.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672980 2024-09-17T19:22:29.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:29.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672980 2024-09-17T19:22:29.412 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672980 2024-09-17T19:22:29.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-17T19:22:29.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:22:29.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-17T19:22:29.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:29.737 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:29.738 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-17T19:22:29.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:22:29.739 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-17T19:22:29.740 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:29.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-17T19:22:29.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-17T19:22:29.741 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509449 2024-09-17T19:22:29.741 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:22:30.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-17T19:22:30.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:22:30.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:22:30.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:22:30.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:22:30.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:22:30.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:22:30.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:22:30.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:22:30.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:22:30.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:22:30.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:22:30.721 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:22:30.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:22:30.722 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:22:30.722 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:22:31.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:22:31.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:22:31.099 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:22:31.099 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:22:31.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:22:31.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:22:31.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:22:31.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:22:31.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:22:31.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:22:31.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:22:31.205 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:22:31.205 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:22:31.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:22:31.504 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:22:31.504 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:22:31.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:22:31.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:31.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:22:31.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-17T19:22:31.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-17T19:22:31.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-17T19:22:31.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:31.677 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:22:31.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-17T19:22:31.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-17T19:22:31.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672982' 2024-09-17T19:22:31.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:31.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:22:31.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-17T19:22:31.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-17T19:22:31.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672982 2-64424509451' 2024-09-17T19:22:31.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:31.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-17T19:22:31.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:31.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:22:31.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-17T19:22:31.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:31.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-17T19:22:31.960 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836506 2024-09-17T19:22:31.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-17T19:22:31.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:22:32.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-17T19:22:32.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:22:33.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:22:33.276 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:22:33.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836507 -lt 21474836506 2024-09-17T19:22:33.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:33.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-17T19:22:33.609 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:33.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:22:33.612 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-17T19:22:33.612 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:33.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-17T19:22:33.613 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672982 2024-09-17T19:22:33.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-17T19:22:33.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:22:33.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-17T19:22:33.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:33.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-17T19:22:33.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:33.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:22:33.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-17T19:22:33.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:33.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-17T19:22:33.961 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509451 2024-09-17T19:22:33.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-17T19:22:33.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:22:34.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-17T19:22:34.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:22:34.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:22:34.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:22:34.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:22:34.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:22:35.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:22:35.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:22:35.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:22:35.062 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:22:35.456 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:22:35.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:22:35.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:22:35.457 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-17T19:22:35.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:22:35.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-17T19:22:35.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-17T19:22:35.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:22:35.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:22:35.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-17T19:22:35.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:22:35.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:22:35.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:22:35.975 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:22:35.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:22:36.578 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:22:36.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:22:36.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:22:36.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:22:36.791 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-17T19:22:36.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:36.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:22:36.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:22:38.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:22:38.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:38.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:22:38.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:22:38.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:22:38.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:22:38.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:22:38.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:40.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:22:40.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:22:40.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:22:40.393 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:22:40.395 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:22:40.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:22:40.397 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:22:40.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:22:40.399 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:22:40.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:22:40.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:22:40.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:22:40.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:22:40.429 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:40.428+0000 7f7bd88987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:22:40.432 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:40.432+0000 7f7bd88987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:22:40.434 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:40.432+0000 7f7bd88987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:22:40.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:22:40.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:22:40.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:40.731 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:22:41.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:41.914 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:41.912+0000 7f7bd88987c0 -1 Falling back to public interface 2024-09-17T19:22:42.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:42.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:42.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:22:42.081 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:22:42.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:42.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:22:42.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:43.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:22:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:22:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:22:43.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:44.663 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:44.660+0000 7f7bd88987c0 -1 osd.1 34 log_to_monitors true 2024-09-17T19:22:44.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:44.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:44.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:22:44.746 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:22:44.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:44.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:22:45.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:46.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:46.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:46.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:22:46.130 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:22:46.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:46.131 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/1527871762,v1:127.0.0.1:6811/1527871762] [v2:127.0.0.1:6812/1527871762,v1:127.0.0.1:6813/1527871762] exists,up 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:22:46.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:22:46.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:22:46.505 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:22:46.505 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:22:46.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:22:46.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:22:46.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:22:46.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:22:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:22:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:22:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:22:46.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:22:46.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:22:47.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:22:47.014 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:22:47.014 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:22:47.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:22:47.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:47.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:22:47.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836510 2024-09-17T19:22:47.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836510 2024-09-17T19:22:47.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510' 2024-09-17T19:22:47.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:47.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:22:47.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724546 2024-09-17T19:22:47.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724546 2024-09-17T19:22:47.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-167503724546' 2024-09-17T19:22:47.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:22:47.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:22:47.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509455 2024-09-17T19:22:47.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509455 2024-09-17T19:22:47.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836510 1-167503724546 2-64424509455' 2024-09-17T19:22:47.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:47.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836510 2024-09-17T19:22:47.461 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:47.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:22:47.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836510 2024-09-17T19:22:47.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:47.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836510 2024-09-17T19:22:47.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-17T19:22:47.464 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836510 2024-09-17T19:22:47.464 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:22:47.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-17T19:22:47.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:22:48.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:22:48.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:22:49.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836510 2024-09-17T19:22:49.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:22:50.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:22:50.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:22:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836510 2024-09-17T19:22:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:50.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724546 2024-09-17T19:22:50.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:50.436 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:22:50.437 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724546 2024-09-17T19:22:50.437 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:50.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724546 2024-09-17T19:22:50.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-09-17T19:22:50.439 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 167503724546 2024-09-17T19:22:50.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:22:50.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724546 -lt 167503724546 2024-09-17T19:22:50.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:22:50.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509455 2024-09-17T19:22:50.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:22:50.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:22:50.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509455 2024-09-17T19:22:50.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:22:50.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509455 2024-09-17T19:22:50.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-17T19:22:50.785 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509455 2024-09-17T19:22:50.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:22:51.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-17T19:22:51.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:22:51.104 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:22:51.104 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:22:51.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:22:51.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:22:51.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:22:51.618 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:22:51.618 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:22:51.618 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:22:51.618 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:22:51.618 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:22:51.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:22:51.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:22:51.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:22:51.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:22:52.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:22:52.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:22:52.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:22:52.909 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:22:52.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:22:53.352 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:22:53.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:22:53.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:22:53.374 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:22:53.592 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-17T19:22:53.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-17T19:22:53.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:22:53.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-17T19:22:53.597 INFO:tasks.workunit.client.0.smithi105.stderr:1+0 records in 2024-09-17T19:22:53.597 INFO:tasks.workunit.client.0.smithi105.stderr:1+0 records out 2024-09-17T19:22:53.597 INFO:tasks.workunit.client.0.smithi105.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000171929 s, 11.9 MB/s 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:22:53.598 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:22:53.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:22:53.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:22:53.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:22:53.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:22:54.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:22:54.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:54.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:22:54.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:22:54.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:22:54.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:22:54.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:22:54.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:22:57.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:22:57.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:22:57.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:22:57.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:22:57.102 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:22:57.103 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:22:57.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:22:57.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:22:57.106 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:22:57.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:22:57.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:22:57.137 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:57.135+0000 7f17f53617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:22:57.140 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:57.139+0000 7f17f53617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:22:57.141 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:57.139+0000 7f17f53617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:22:57.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:57.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:22:57.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:22:58.434 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:22:58.431+0000 7f17f53617c0 -1 Falling back to public interface 2024-09-17T19:22:58.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:22:58.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:22:58.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:22:58.836 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:22:58.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:22:58.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:22:59.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:00.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:00.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:00.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:23:00.172 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:23:00.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:00.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:00.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:01.157 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:01.155+0000 7f17f53617c0 -1 osd.0 43 log_to_monitors true 2024-09-17T19:23:01.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:01.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:01.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:23:01.496 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:23:01.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:01.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 48 up_thru 39 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/3494064920,v1:127.0.0.1:6803/3494064920] [v2:127.0.0.1:6804/3494064920,v1:127.0.0.1:6805/3494064920] exists,up 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:23:02.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:23:02.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:23:02.008 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:23:02.008 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:23:02.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:23:02.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:23:02.009 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:23:02.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:23:02.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:23:02.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:23:02.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:23:02.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:23:02.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:23:02.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:23:02.692 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:02.692 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:23:02.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:23:02.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:02.692 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:23:02.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430210 2024-09-17T19:23:02.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430210 2024-09-17T19:23:02.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430210' 2024-09-17T19:23:02.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:02.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:23:03.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724550 2024-09-17T19:23:03.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724550 2024-09-17T19:23:03.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430210 1-167503724550' 2024-09-17T19:23:03.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:03.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:23:03.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-17T19:23:03.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-17T19:23:03.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430210 1-167503724550 2-64424509459' 2024-09-17T19:23:03.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:03.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:03.346 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-206158430210 2024-09-17T19:23:03.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:23:03.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-206158430210 2024-09-17T19:23:03.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:03.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430210 2024-09-17T19:23:03.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-09-17T19:23:03.349 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 206158430210 2024-09-17T19:23:03.349 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:23:03.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430210 -lt 206158430210 2024-09-17T19:23:03.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:03.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-167503724550 2024-09-17T19:23:03.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:03.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:23:03.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-167503724550 2024-09-17T19:23:03.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724550 2024-09-17T19:23:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 167503724550' 2024-09-17T19:23:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 167503724550 2024-09-17T19:23:03.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:23:04.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724550 -lt 167503724550 2024-09-17T19:23:04.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:04.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-17T19:23:04.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:04.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:23:04.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-17T19:23:04.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:04.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-17T19:23:04.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-17T19:23:04.037 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509459 2024-09-17T19:23:04.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:23:04.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509459 2024-09-17T19:23:04.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:23:05.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:23:05.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:23:05.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509459 2024-09-17T19:23:05.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:23:05.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:05.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:06.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:23:06.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:06.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:06.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:06.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:06.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:06.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:06.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:06.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:23:06.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:06.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:06.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:23:06.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:23:06.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:23:07.215 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:23:07.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:23:07.809 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:23:07.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:23:07.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:23:07.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:23:08.152 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-17T19:23:08.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:23:08.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:23:08.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-17T19:23:08.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:23:08.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:23:08.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:23:08.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:23:08.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:23:08.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:23:09.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:23:09.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-17T19:23:10.866 INFO:tasks.workunit.client.0.smithi105.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-17T19:23:11.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:23:11.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:23:11.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:23:11.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:23:11.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:23:11.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:23:11.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:23:11.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:23:11.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:23:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:23:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:23:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:23:11.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:23:11.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:23:11.416 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:23:11.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:23:11.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:23:11.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:23:11.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:23:11.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:23:11.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:23:11.448 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:11.447+0000 7fb0fbc4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:11.451 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:11.447+0000 7fb0fbc4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:11.452 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:11.451+0000 7fb0fbc4a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:11.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:23:12.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:12.534 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:12.531+0000 7fb0fbc4a7c0 -1 Falling back to public interface 2024-09-17T19:23:13.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:13.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:13.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:23:13.067 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:13.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:13.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:23:13.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:14.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:23:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:23:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:14.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:23:14.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:15.347 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:15.343+0000 7fb0fbc4a7c0 -1 osd.1 52 log_to_monitors true 2024-09-17T19:23:15.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:15.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:15.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:23:15.743 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:23:15.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:15.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:23:16.101 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 57 up_thru 48 down_at 53 last_clean_interval [39,52) [v2:127.0.0.1:6810/330699853,v1:127.0.0.1:6811/330699853] [v2:127.0.0.1:6812/330699853,v1:127.0.0.1:6813/330699853] exists,up 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:23:16.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:23:16.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:23:16.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:23:16.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:23:16.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:23:16.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:23:16.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:23:16.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:23:16.214 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:23:16.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:23:16.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:23:16.588 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:16.589 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:23:16.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:23:16.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:16.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:23:16.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=206158430214 2024-09-17T19:23:16.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 206158430214 2024-09-17T19:23:16.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430214' 2024-09-17T19:23:16.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:16.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:23:17.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135874 2024-09-17T19:23:17.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135874 2024-09-17T19:23:17.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430214 1-244813135874' 2024-09-17T19:23:17.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:17.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:23:17.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509463 2024-09-17T19:23:17.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509463 2024-09-17T19:23:17.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-206158430214 1-244813135874 2-64424509463' 2024-09-17T19:23:17.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:17.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-206158430214 2024-09-17T19:23:17.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:17.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:23:17.464 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-206158430214 2024-09-17T19:23:17.464 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:17.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=206158430214 2024-09-17T19:23:17.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 206158430214' 2024-09-17T19:23:17.466 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 206158430214 2024-09-17T19:23:17.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:23:17.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 206158430214 -lt 206158430214 2024-09-17T19:23:17.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:17.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135874 2024-09-17T19:23:17.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:17.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:23:17.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135874 2024-09-17T19:23:17.829 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:17.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135874 2024-09-17T19:23:17.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-09-17T19:23:17.831 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 244813135874 2024-09-17T19:23:17.831 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:23:18.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135874 -lt 244813135874 2024-09-17T19:23:18.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:18.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509463 2024-09-17T19:23:18.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:18.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:23:18.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509463 2024-09-17T19:23:18.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:18.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509463 2024-09-17T19:23:18.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-09-17T19:23:18.155 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509463 2024-09-17T19:23:18.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:23:18.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509463 2024-09-17T19:23:18.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:23:19.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:23:19.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:23:19.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509463 -lt 64424509463 2024-09-17T19:23:19.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:23:19.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:19.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:20.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:23:20.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:20.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:20.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:20.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:20.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:20.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:20.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:20.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:23:20.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:20.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:20.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:20.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:23:20.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:23:20.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:23:20.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:23:20.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:23:20.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:23:21.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:23:21.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:21.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:21.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:21.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:21.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:21.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:21.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:23:21.431 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:21.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:21.432 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:21.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:23:21.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 3 2024-09-17T19:23:21.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:23:21.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-17T19:23:21.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:23:21.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:23:21.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:23:21.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:21.912 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:22.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:23:22.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:22.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:22.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:22.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:23:22.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:23:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:23:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:23:23.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:23:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:23:23.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:23:23.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:23:23.242 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:23.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:23.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:23.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:23.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:23.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:23.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:23.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:23:23.567 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:23.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:23.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:23.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:23:23.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:23:23.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:23:24.424 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:23:24.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:23:24.993 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:23:25.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:23:25.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:23:25.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:23:25.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-17T19:23:25.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-17T19:23:25.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:23:25.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-17T19:23:25.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:23:25.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:23:25.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:23:26.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:23:28.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:23:28.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:23:28.982 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:23:28.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:23:28.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:23:28.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:23:28.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:23:28.986 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:23:28.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:23:28.986 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:23:28.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:23:28.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:23:28.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:23:28.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:23:29.018 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:29.015+0000 7f2f757ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:29.020 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:29.019+0000 7f2f757ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:29.022 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:29.019+0000 7f2f757ad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:29.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:23:29.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:23:29.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:23:29.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:23:29.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:23:29.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:29.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:23:29.310 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:23:29.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:29.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:29.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:30.510 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:30.507+0000 7f2f757ad7c0 -1 Falling back to public interface 2024-09-17T19:23:30.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:30.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:23:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:30.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:31.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:31.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:31.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:23:31.976 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:23:31.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:31.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:32.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:33.202 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:33.199+0000 7f2f757ad7c0 -1 osd.0 62 log_to_monitors true 2024-09-17T19:23:33.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:33.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:33.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:23:33.313 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:23:33.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:33.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:33.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:34.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:34.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:34.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:23:34.736 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:23:34.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:34.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:23:35.077 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [48,62) [v2:127.0.0.1:6802/514737193,v1:127.0.0.1:6803/514737193] [v2:127.0.0.1:6804/514737193,v1:127.0.0.1:6805/514737193] exists,up 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:23:35.078 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:23:35.079 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:23:35.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:23:35.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:23:35.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:23:35.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:23:35.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:23:35.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:23:35.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:23:35.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:23:35.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:23:35.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:23:35.695 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:35.696 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:23:35.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:23:35.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:35.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:23:35.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808834 2024-09-17T19:23:35.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808834 2024-09-17T19:23:35.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834' 2024-09-17T19:23:35.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:35.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:23:35.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135879 2024-09-17T19:23:35.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135879 2024-09-17T19:23:35.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-244813135879' 2024-09-17T19:23:35.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:35.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:23:36.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509468 2024-09-17T19:23:36.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509468 2024-09-17T19:23:36.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808834 1-244813135879 2-64424509468' 2024-09-17T19:23:36.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:36.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808834 2024-09-17T19:23:36.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:36.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:23:36.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808834 2024-09-17T19:23:36.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:36.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808834 2024-09-17T19:23:36.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-09-17T19:23:36.125 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 287762808834 2024-09-17T19:23:36.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:23:36.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 287762808834 2024-09-17T19:23:36.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:23:37.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:23:37.433 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:23:37.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808834 -lt 287762808834 2024-09-17T19:23:37.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:37.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135879 2024-09-17T19:23:37.764 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:37.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:23:37.766 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135879 2024-09-17T19:23:37.767 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:37.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135879 2024-09-17T19:23:37.768 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135879' 2024-09-17T19:23:37.768 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 244813135879 2024-09-17T19:23:37.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:23:38.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135879 -lt 244813135879 2024-09-17T19:23:38.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:38.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509468 2024-09-17T19:23:38.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:38.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:23:38.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509468 2024-09-17T19:23:38.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:38.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509468 2024-09-17T19:23:38.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-17T19:23:38.099 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509468 2024-09-17T19:23:38.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:23:38.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509468 -lt 64424509468 2024-09-17T19:23:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:23:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:38.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:38.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:23:38.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:38.822 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:39.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:23:39.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:39.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:39.172 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:39.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:23:39.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:23:39.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T19:23:39.542 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:23:39.542 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:23:39.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:23:39.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:23:39.542 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:23:40.854 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:23:43.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T19:23:43.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:23:43.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:23:43.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T19:23:43.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:23:43.078 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:23:43.080 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T19:23:43.081 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.2 2024-09-17T19:23:43.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:23:43.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T19:23:43.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T19:23:43.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:23:43.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:23:43.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:23:43.116 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:43.114+0000 7fc382c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:43.118 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:43.114+0000 7fc382c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:43.120 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:43.118+0000 7fc382c797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:43.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T19:23:43.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:23:43.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:23:43.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:23:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:23:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:23:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:23:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:43.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:44.402 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:44.398+0000 7fc382c797c0 -1 Falling back to public interface 2024-09-17T19:23:44.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:44.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:44.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:23:44.773 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:44.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:44.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:45.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:46.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:46.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:46.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:23:46.101 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:23:46.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:46.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:46.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:47.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:47.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:47.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:23:47.435 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:23:47.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:47.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:47.436 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:47.434+0000 7fc382c797c0 -1 osd.2 69 log_to_monitors true 2024-09-17T19:23:47.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:48.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:48.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:48.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:23:48.786 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:23:48.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:48.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:49.303 INFO:tasks.workunit.client.0.smithi105.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/77445976,v1:127.0.0.1:6819/77445976] [v2:127.0.0.1:6820/77445976,v1:127.0.0.1:6821/77445976] exists,up 6b05f872-817c-4060-aa7f-3ccd7598ee4e 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:23:49.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:23:49.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:23:49.305 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:23:49.305 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:23:49.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:23:49.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:23:49.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:23:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:23:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:23:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:23:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:23:49.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:23:49.418 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:23:49.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:23:49.882 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:49.882 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:23:49.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:23:49.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:49.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:23:50.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808838 2024-09-17T19:23:50.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808838 2024-09-17T19:23:50.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808838' 2024-09-17T19:23:50.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:50.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:23:50.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135882 2024-09-17T19:23:50.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135882 2024-09-17T19:23:50.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808838 1-244813135882' 2024-09-17T19:23:50.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:23:50.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:23:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=313532612610 2024-09-17T19:23:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 313532612610 2024-09-17T19:23:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808838 1-244813135882 2-313532612610' 2024-09-17T19:23:50.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:50.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808838 2024-09-17T19:23:50.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:50.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:23:50.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808838 2024-09-17T19:23:50.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:50.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808838 2024-09-17T19:23:50.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808838' 2024-09-17T19:23:50.333 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 287762808838 2024-09-17T19:23:50.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:23:50.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808837 -lt 287762808838 2024-09-17T19:23:50.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:23:51.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:23:51.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:23:51.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808838 -lt 287762808838 2024-09-17T19:23:51.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:51.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135882 2024-09-17T19:23:51.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:51.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:23:51.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135882 2024-09-17T19:23:51.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:51.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135882 2024-09-17T19:23:51.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135882' 2024-09-17T19:23:51.991 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 244813135882 2024-09-17T19:23:51.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:23:52.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135883 -lt 244813135882 2024-09-17T19:23:52.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:23:52.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-313532612610 2024-09-17T19:23:52.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:23:52.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:23:52.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-313532612610 2024-09-17T19:23:52.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:23:52.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=313532612610 2024-09-17T19:23:52.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-09-17T19:23:52.330 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 313532612610 2024-09-17T19:23:52.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:23:52.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 313532612610 -lt 313532612610 2024-09-17T19:23:52.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:23:52.653 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:52.653 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:53.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:23:53.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:23:53.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:23:53.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:23:53.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:23:53.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:23:53.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:23:53.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:23:53.399 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:23:53.400 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:23:53.400 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:23:53.400 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:23:53.779 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:23:53.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:23:53.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:23:53.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:23:53.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:23:54.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:23:54.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:23:54.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:23:54.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:23:54.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T19:23:54.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:23:54.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:23:54.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:23:56.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T19:23:56.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:23:56.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:23:56.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:23:56.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:23:56.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:23:56.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T19:23:56.659 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.2 2024-09-17T19:23:56.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:23:56.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T19:23:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T19:23:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:23:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:23:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:23:56.693 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:56.690+0000 7f6c5b86e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:56.696 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:56.694+0000 7f6c5b86e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:56.697 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:56.694+0000 7f6c5b86e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:23:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T19:23:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:23:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:23:56.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:23:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:23:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:23:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:23:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:56.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:57.330 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:57.990 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:23:57.986+0000 7f6c5b86e7c0 -1 Falling back to public interface 2024-09-17T19:23:58.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:58.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:58.332 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:23:58.333 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:23:58.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:58.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:58.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:23:59.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:23:59.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:23:59.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:23:59.650 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:23:59.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:23:59.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:23:59.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:00.636 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:00.634+0000 7f6c5b86e7c0 -1 osd.2 75 log_to_monitors true 2024-09-17T19:24:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:00.968 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:24:00.969 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:24:00.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:00.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:osd.2 up in weight 1 up_from 80 up_thru 73 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/2988778637,v1:127.0.0.1:6819/2988778637] [v2:127.0.0.1:6820/2988778637,v1:127.0.0.1:6821/2988778637] exists,up 6b05f872-817c-4060-aa7f-3ccd7598ee4e 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:24:01.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:24:01.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:24:01.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:24:01.471 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:24:01.472 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:24:01.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:24:01.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:24:01.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:24:01.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:24:01.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:24:01.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:24:01.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:24:01.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:24:01.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:24:01.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:24:01.973 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:01.973 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:24:01.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:24:01.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:01.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:24:02.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808841 2024-09-17T19:24:02.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808841 2024-09-17T19:24:02.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841' 2024-09-17T19:24:02.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:02.205 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:24:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=244813135886 2024-09-17T19:24:02.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 244813135886 2024-09-17T19:24:02.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-244813135886' 2024-09-17T19:24:02.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:02.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:24:02.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383682 2024-09-17T19:24:02.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383682 2024-09-17T19:24:02.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808841 1-244813135886 2-343597383682' 2024-09-17T19:24:02.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:02.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808841 2024-09-17T19:24:02.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:02.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:24:02.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808841 2024-09-17T19:24:02.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:02.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808841 2024-09-17T19:24:02.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808841' 2024-09-17T19:24:02.651 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 287762808841 2024-09-17T19:24:02.651 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:02.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808840 -lt 287762808841 2024-09-17T19:24:02.970 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:24:03.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:24:03.973 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:04.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808841 -lt 287762808841 2024-09-17T19:24:04.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:04.300 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-244813135886 2024-09-17T19:24:04.300 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:04.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:24:04.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-244813135886 2024-09-17T19:24:04.302 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:04.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=244813135886 2024-09-17T19:24:04.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 244813135886' 2024-09-17T19:24:04.304 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 244813135886 2024-09-17T19:24:04.305 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:24:04.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 244813135886 -lt 244813135886 2024-09-17T19:24:04.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:04.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383682 2024-09-17T19:24:04.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:04.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:24:04.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383682 2024-09-17T19:24:04.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:04.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383682 2024-09-17T19:24:04.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-09-17T19:24:04.634 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 343597383682 2024-09-17T19:24:04.635 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:24:04.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383682 -lt 343597383682 2024-09-17T19:24:04.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:24:04.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:04.957 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:05.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:24:05.350 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:24:05.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:24:05.702 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:24:05.702 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:05.702 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:06.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:24:06.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:24:06.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:24:06.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-17T19:24:06.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:24:06.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-17T19:24:06.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-17T19:24:06.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:24:06.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:24:06.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-17T19:24:06.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:24:06.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:24:06.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:24:06.536 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:24:06.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:24:07.121 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:24:07.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:24:07.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:24:07.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:24:07.435 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-17T19:24:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:24:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:24:07.437 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:2+0 records in 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:2+0 records out 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000188534 s, 21.7 MB/s 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:24:07.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:24:07.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:24:07.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:24:07.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:24:10.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:24:10.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:24:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:24:10.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:24:10.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:24:10.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:24:10.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:24:10.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:24:10.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:24:10.576 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:24:10.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:24:10.577 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:24:10.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:24:10.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:24:10.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:24:10.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:24:10.607 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:10.602+0000 7efced9d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:10.610 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:10.606+0000 7efced9d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:10.611 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:10.610+0000 7efced9d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:10.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:24:10.918 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:24:10.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:10.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:11.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:12.134 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:12.134+0000 7efced9d77c0 -1 Falling back to public interface 2024-09-17T19:24:12.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:12.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:24:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:12.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:12.598 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:13.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:13.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:13.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:24:13.602 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:24:13.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:13.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:13.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:14.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:14.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:14.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:24:14.929 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:24:14.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:14.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:15.028 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:15.030+0000 7efced9d77c0 -1 osd.1 84 log_to_monitors true 2024-09-17T19:24:15.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:16.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:16.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:16.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:24:16.347 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:24:16.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:16.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:16.680 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/2366604110,v1:127.0.0.1:6811/2366604110] [v2:127.0.0.1:6812/2366604110,v1:127.0.0.1:6813/2366604110] exists,up 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:24:16.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:24:16.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:24:16.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:24:16.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:24:16.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:24:16.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:24:16.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:24:16.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:24:16.683 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:24:16.683 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:24:16.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:24:16.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:24:16.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:24:16.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:24:16.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:24:16.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:24:16.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:24:16.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:24:16.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:24:17.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:24:17.146 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:17.146 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:24:17.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:24:17.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:17.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:24:17.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=287762808845 2024-09-17T19:24:17.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 287762808845 2024-09-17T19:24:17.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845' 2024-09-17T19:24:17.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:17.296 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:24:17.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089346 2024-09-17T19:24:17.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089346 2024-09-17T19:24:17.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845 1-382252089346' 2024-09-17T19:24:17.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:17.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:24:17.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383686 2024-09-17T19:24:17.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383686 2024-09-17T19:24:17.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-287762808845 1-382252089346 2-343597383686' 2024-09-17T19:24:17.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:17.581 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:17.581 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-287762808845 2024-09-17T19:24:17.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:24:17.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-287762808845 2024-09-17T19:24:17.583 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:17.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=287762808845 2024-09-17T19:24:17.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 287762808845' 2024-09-17T19:24:17.584 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 287762808845 2024-09-17T19:24:17.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808844 -lt 287762808845 2024-09-17T19:24:17.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:24:18.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:24:18.905 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:19.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808844 -lt 287762808845 2024-09-17T19:24:19.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:24:20.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:24:20.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:20.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 287762808846 -lt 287762808845 2024-09-17T19:24:20.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:20.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089346 2024-09-17T19:24:20.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:20.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:24:20.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089346 2024-09-17T19:24:20.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:20.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089346 2024-09-17T19:24:20.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-09-17T19:24:20.555 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 382252089346 2024-09-17T19:24:20.555 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:24:20.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089346 -lt 382252089346 2024-09-17T19:24:20.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:20.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383686 2024-09-17T19:24:20.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:20.900 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:24:20.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383686 2024-09-17T19:24:20.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:20.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383686 2024-09-17T19:24:20.903 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383686' 2024-09-17T19:24:20.903 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 343597383686 2024-09-17T19:24:20.903 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:24:21.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383686 -lt 343597383686 2024-09-17T19:24:21.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:24:21.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:21.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:21.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:24:21.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:24:21.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:24:21.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:24:21.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:24:21.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:24:21.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:24:21.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:24:21.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:24:21.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:24:21.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:21.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:22.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:24:22.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:24:22.825 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:24:22.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:24:23.276 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:24:23.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:24:23.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:24:23.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:24:23.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-17T19:24:23.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-17T19:24:23.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:24:23.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:24:23.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:24:23.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:24:23.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:24:23.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:24:23.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:24:23.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:24:25.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:24:27.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:24:27.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:24:27.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:24:27.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:24:27.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:24:27.136 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:24:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:24:27.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:24:27.138 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:24:27.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:24:27.139 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:24:27.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:24:27.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:24:27.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:24:27.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:24:27.173 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:27.173+0000 7fc40c7c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:27.175 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:27.177+0000 7fc40c7c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:27.177 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:27.177+0000 7fc40c7c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:24:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:27.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:27.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:28.454 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:28.453+0000 7fc40c7c07c0 -1 Falling back to public interface 2024-09-17T19:24:28.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:28.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:28.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:24:28.789 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:28.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:28.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:29.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:30.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:30.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:30.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:24:30.128 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:24:30.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:30.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:30.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:31.276 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:31.277+0000 7fc40c7c07c0 -1 osd.0 93 log_to_monitors true 2024-09-17T19:24:31.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:31.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:31.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:24:31.459 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:24:31.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:31.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:31.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:32.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:32.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:32.955 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:24:32.955 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:24:32.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:32.956 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [67,93) [v2:127.0.0.1:6802/4150284942,v1:127.0.0.1:6803/4150284942] [v2:127.0.0.1:6804/4150284942,v1:127.0.0.1:6805/4150284942] exists,up 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:24:33.480 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:24:33.481 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:24:33.481 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:24:33.481 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:24:33.482 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:24:33.482 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:24:33.482 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:24:33.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:24:33.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:24:33.600 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:24:33.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:24:33.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:24:33.601 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:24:33.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:24:33.928 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:33.928 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:24:33.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:24:33.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:33.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:24:34.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795010 2024-09-17T19:24:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795010 2024-09-17T19:24:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010' 2024-09-17T19:24:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:34.085 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:24:34.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=382252089350 2024-09-17T19:24:34.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 382252089350 2024-09-17T19:24:34.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010 1-382252089350' 2024-09-17T19:24:34.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:34.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:24:34.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383690 2024-09-17T19:24:34.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383690 2024-09-17T19:24:34.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795010 1-382252089350 2-343597383690' 2024-09-17T19:24:34.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:34.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-420906795010 2024-09-17T19:24:34.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:34.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:24:34.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-420906795010 2024-09-17T19:24:34.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795010 2024-09-17T19:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-09-17T19:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 420906795010 2024-09-17T19:24:34.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:34.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 420906795010 2024-09-17T19:24:34.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:24:35.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:24:35.686 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:36.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795010 -lt 420906795010 2024-09-17T19:24:36.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:36.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-382252089350 2024-09-17T19:24:36.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:36.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:24:36.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-382252089350 2024-09-17T19:24:36.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:36.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=382252089350 2024-09-17T19:24:36.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2024-09-17T19:24:36.026 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 382252089350 2024-09-17T19:24:36.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:24:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 382252089350 -lt 382252089350 2024-09-17T19:24:36.354 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:36.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:36.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383690 2024-09-17T19:24:36.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:24:36.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383690 2024-09-17T19:24:36.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:36.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383690 2024-09-17T19:24:36.359 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383690' 2024-09-17T19:24:36.359 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 343597383690 2024-09-17T19:24:36.360 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:24:36.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383690 -lt 343597383690 2024-09-17T19:24:36.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:24:36.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:36.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:37.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:24:37.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:24:37.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:24:37.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:24:37.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:24:37.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:24:37.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:24:37.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:24:37.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:24:37.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:24:37.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:37.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:37.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:24:37.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:24:37.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:24:37.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:24:37.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:24:37.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:24:37.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:24:37.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:24:37.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:24:37.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:24:37.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:24:38.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:24:38.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:24:40.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:24:40.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:24:40.604 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:24:40.604 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:24:40.604 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:24:40.604 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:24:40.604 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:24:40.606 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:24:40.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:24:40.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:24:40.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:24:40.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:24:40.639 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:40.637+0000 7f523da477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:40.642 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:40.641+0000 7f523da477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:40.644 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:40.645+0000 7f523da477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:40.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:24:40.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:40.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:41.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:41.706 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:41.705+0000 7f523da477c0 -1 Falling back to public interface 2024-09-17T19:24:42.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:42.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:42.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:24:42.304 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:42.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:42.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:42.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:43.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:43.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:43.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:24:43.622 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:24:43.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:43.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:43.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:44.456 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:44.457+0000 7f523da477c0 -1 osd.1 100 log_to_monitors true 2024-09-17T19:24:44.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:44.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:44.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:24:44.944 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:24:44.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:44.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:24:45.702 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 105 up_thru 105 down_at 101 last_clean_interval [89,100) [v2:127.0.0.1:6810/2241874813,v1:127.0.0.1:6811/2241874813] [v2:127.0.0.1:6812/2241874813,v1:127.0.0.1:6813/2241874813] exists,up 1052c18e-53c6-42d0-9f5f-81716a682fc7 2024-09-17T19:24:45.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:24:45.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:24:45.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:24:45.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:24:45.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:24:45.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:24:45.703 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:24:45.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:24:45.704 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:24:45.704 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:24:45.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:24:45.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:24:45.704 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:24:45.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:24:45.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:24:45.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:24:45.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:24:45.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:24:45.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:24:46.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:24:46.179 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:46.179 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:24:46.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:24:46.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:46.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:24:46.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=420906795013 2024-09-17T19:24:46.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 420906795013 2024-09-17T19:24:46.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795013' 2024-09-17T19:24:46.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:46.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:24:46.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566082 2024-09-17T19:24:46.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566082 2024-09-17T19:24:46.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795013 1-450971566082' 2024-09-17T19:24:46.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:24:46.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:24:46.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383694 2024-09-17T19:24:46.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383694 2024-09-17T19:24:46.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-420906795013 1-450971566082 2-343597383694' 2024-09-17T19:24:46.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:46.634 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-420906795013 2024-09-17T19:24:46.634 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:46.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:24:46.635 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-420906795013 2024-09-17T19:24:46.636 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:46.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=420906795013 2024-09-17T19:24:46.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 420906795013' 2024-09-17T19:24:46.637 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 420906795013 2024-09-17T19:24:46.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:46.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795012 -lt 420906795013 2024-09-17T19:24:46.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:24:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:24:47.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:24:48.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 420906795014 -lt 420906795013 2024-09-17T19:24:48.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:48.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566082 2024-09-17T19:24:48.284 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:48.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:24:48.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566082 2024-09-17T19:24:48.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:48.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566082 2024-09-17T19:24:48.287 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566082' 2024-09-17T19:24:48.287 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 450971566082 2024-09-17T19:24:48.288 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:24:48.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566082 -lt 450971566082 2024-09-17T19:24:48.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:24:48.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:24:48.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383694 2024-09-17T19:24:48.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:24:48.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:24:48.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383694 2024-09-17T19:24:48.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383694 2024-09-17T19:24:48.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383694' 2024-09-17T19:24:48.618 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 343597383694 2024-09-17T19:24:48.619 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:24:48.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383694 -lt 343597383694 2024-09-17T19:24:48.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:24:48.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:48.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:49.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:24:49.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:24:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:24:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:24:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:24:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:24:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:24:49.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:24:49.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:24:49.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:24:49.731 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:24:49.731 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:24:50.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-17T19:24:50.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:24:50.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:24:50.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:24:50.568 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:24:50.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:24:51.128 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:24:51.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:24:51.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:24:51.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:24:51.333 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-17T19:24:51.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:24:51.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:24:51.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-17T19:24:51.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-17T19:24:51.335 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-17T19:24:51.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-17T19:24:51.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:24:51.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:24:51.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:24:51.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:24:51.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:24:51.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:24:51.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:24:51.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:24:51.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:24:52.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:24:52.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-17T19:24:55.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:24:55.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:24:55.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:24:55.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:24:55.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:24:55.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:24:55.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:24:55.024 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:24:55.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:24:55.025 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:24:55.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:24:55.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:24:55.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:24:55.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:24:55.062 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:55.061+0000 7f58dae5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:55.066 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:55.065+0000 7f58dae5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:55.067 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:55.065+0000 7f58dae5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:24:55.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:24:55.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:55.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:55.714 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:56.615 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:56.613+0000 7f58dae5d7c0 -1 Falling back to public interface 2024-09-17T19:24:56.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:56.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:56.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:24:56.716 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:24:56.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:56.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:57.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:58.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:58.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:58.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:24:58.054 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:24:58.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:58.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:58.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:24:59.330 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:24:59.329+0000 7f58dae5d7c0 -1 osd.0 109 log_to_monitors true 2024-09-17T19:24:59.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:24:59.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:24:59.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:24:59.381 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:24:59.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:24:59.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:24:59.846 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 114 up_thru 105 down_at 110 last_clean_interval [98,109) [v2:127.0.0.1:6802/684432161,v1:127.0.0.1:6803/684432161] [v2:127.0.0.1:6804/684432161,v1:127.0.0.1:6805/684432161] exists,up 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:24:59.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:24:59.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:24:59.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:24:59.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:24:59.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:24:59.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:24:59.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:24:59.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:24:59.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:24:59.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:25:00.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:25:00.349 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:25:00.349 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:25:00.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:25:00.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:25:00.349 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:25:00.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=489626271746 2024-09-17T19:25:00.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 489626271746 2024-09-17T19:25:00.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-489626271746' 2024-09-17T19:25:00.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:25:00.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:25:00.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566086 2024-09-17T19:25:00.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566086 2024-09-17T19:25:00.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-489626271746 1-450971566086' 2024-09-17T19:25:00.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:25:00.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:25:01.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383698 2024-09-17T19:25:01.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383698 2024-09-17T19:25:01.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-489626271746 1-450971566086 2-343597383698' 2024-09-17T19:25:01.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:25:01.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-489626271746 2024-09-17T19:25:01.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:25:01.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:25:01.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-489626271746 2024-09-17T19:25:01.150 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:25:01.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=489626271746 2024-09-17T19:25:01.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 489626271746' 2024-09-17T19:25:01.151 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 489626271746 2024-09-17T19:25:01.152 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:25:01.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 489626271746 -lt 489626271746 2024-09-17T19:25:01.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:25:01.501 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566086 2024-09-17T19:25:01.501 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:25:01.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:25:01.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566086 2024-09-17T19:25:01.503 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:25:01.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566086 2024-09-17T19:25:01.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566086' 2024-09-17T19:25:01.505 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 450971566086 2024-09-17T19:25:01.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:25:01.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566086 -lt 450971566086 2024-09-17T19:25:01.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:25:01.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:25:01.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383698 2024-09-17T19:25:01.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:25:01.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383698 2024-09-17T19:25:01.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:25:01.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383698 2024-09-17T19:25:01.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383698' 2024-09-17T19:25:01.843 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 343597383698 2024-09-17T19:25:01.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:25:02.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383698 -lt 343597383698 2024-09-17T19:25:02.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:25:02.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:02.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:02.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:25:02.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:25:02.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:25:02.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:25:02.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:25:02.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:25:02.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:25:02.553 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:25:02.901 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:25:02.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:25:02.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:02.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:25:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T19:25:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:25:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T19:25:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:25:03.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:25:03.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:25:03.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:25:03.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:25:03.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:25:03.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:25:03.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:25:03.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:25:03.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:25:03.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:25:03.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:25:03.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:03.701 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:04.067 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:25:04.068 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:25:04.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:25:04.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:25:04.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:25:04.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:25:04.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:25:04.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:25:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:25:04.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:25:04.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:25:04.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:25:04.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:25:04.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:25:04.667 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:25:04.667 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:25:04.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:25:04.975 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:25:04.975 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:04.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:25:05.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:25:05.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:25:05.352 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:25:05.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:25:05.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:25:05.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-17T19:25:05.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:25:05.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T19:25:06.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:25:06.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:25:06.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:25:06.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:25:06.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:25:06.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:25:06.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:25:06.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:25:06.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:25:06.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:25:06.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:06.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:06.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:25:06.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 0 2024-09-17T19:25:06.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:25:06.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:25:06.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:25:06.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:25:06.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:25:06.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:25:06.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:25:06.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:25:06.984 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:25:06.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:25:06.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:25:06.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:25:07.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:25:07.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:25:07.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:07.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:07.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:25:07.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:25:07.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:25:07.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:25:07.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:25:07.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:25:07.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:25:07.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:25:07.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:25:07.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:25:07.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:25:08.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:25:08.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:25:08.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:25:08.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:25:08.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:25:08.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:25:08.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:25:08.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:25:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:25:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:25:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:25:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:25:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:25:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:25:10.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:25:10.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:25:10.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:25:10.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:25:10.932 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:25:10.932 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:10.932 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:25:10.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:25:10.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:25:10.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:25:10.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:25:10.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:25:10.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:25:10.966 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:10.964+0000 7f272e7627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:10.969 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:10.968+0000 7f272e7627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:10.970 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:10.968+0000 7f272e7627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:11.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:25:11.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:25:11.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:25:11.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:25:11.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:25:11.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:11.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:25:11.268 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:25:11.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:11.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:11.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:12.274 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:12.272+0000 7f272e7627c0 -1 Falling back to public interface 2024-09-17T19:25:12.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:12.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:12.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:25:12.580 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:25:12.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:12.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:12.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:13.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:13.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:13.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:25:13.916 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:25:13.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:13.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:14.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:15.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:15.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:15.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:25:15.229 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:25:15.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:15.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:15.324 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:15.324+0000 7f272e7627c0 -1 osd.0 116 log_to_monitors true 2024-09-17T19:25:15.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:16.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:16.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:16.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:25:16.584 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:25:16.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:16.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:17.116 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 121 up_thru 121 down_at 117 last_clean_interval [114,116) [v2:127.0.0.1:6802/2123901118,v1:127.0.0.1:6803/2123901118] [v2:127.0.0.1:6804/2123901118,v1:127.0.0.1:6805/2123901118] exists,up 7e026281-87cb-482c-9e9d-db7121e47cfe 2024-09-17T19:25:17.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:25:17.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:25:17.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:25:17.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:25:17.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:25:17.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:25:17.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:25:17.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:25:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:25:17.118 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:25:17.118 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:25:17.118 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:25:17.118 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:25:17.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:25:17.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:25:17.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:25:17.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:25:17.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:25:17.231 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:25:17.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:25:17.668 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:25:17.668 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:25:17.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:25:17.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:25:17.668 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:25:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=519691042818 2024-09-17T19:25:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 519691042818 2024-09-17T19:25:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-519691042818' 2024-09-17T19:25:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:25:17.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:25:17.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=450971566090 2024-09-17T19:25:17.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 450971566090 2024-09-17T19:25:17.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-519691042818 1-450971566090' 2024-09-17T19:25:17.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:25:17.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:25:18.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=343597383702 2024-09-17T19:25:18.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 343597383702 2024-09-17T19:25:18.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-519691042818 1-450971566090 2-343597383702' 2024-09-17T19:25:18.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:25:18.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-519691042818 2024-09-17T19:25:18.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:25:18.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:25:18.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-519691042818 2024-09-17T19:25:18.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:25:18.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=519691042818 2024-09-17T19:25:18.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 519691042818' 2024-09-17T19:25:18.116 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 519691042818 2024-09-17T19:25:18.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:25:18.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 519691042818 2024-09-17T19:25:18.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:25:19.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:25:19.433 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:25:19.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 519691042818 -lt 519691042818 2024-09-17T19:25:19.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:25:19.771 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:25:19.771 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-450971566090 2024-09-17T19:25:19.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:25:19.773 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:25:19.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-450971566090 2024-09-17T19:25:19.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=450971566090 2024-09-17T19:25:19.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 450971566090' 2024-09-17T19:25:19.775 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 450971566090 2024-09-17T19:25:19.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:25:20.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 450971566090 -lt 450971566090 2024-09-17T19:25:20.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:25:20.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:25:20.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-343597383702 2024-09-17T19:25:20.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:25:20.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-343597383702 2024-09-17T19:25:20.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:25:20.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=343597383702 2024-09-17T19:25:20.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 343597383702' 2024-09-17T19:25:20.139 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 343597383702 2024-09-17T19:25:20.140 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:25:20.466 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 343597383702 -lt 343597383702 2024-09-17T19:25:20.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:25:20.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:20.468 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:20.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:25:20.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:25:21.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:25:21.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:25:21.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:25:21.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:25:21.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:25:21.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:25:21.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:25:21.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-17T19:25:21.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-17T19:25:21.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:25:21.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-17T19:25:21.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-17T19:25:21.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:25:21.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-17T19:25:21.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-17T19:25:21.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-17T19:25:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-17T19:25:21.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:25:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:21.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:25:22.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:22.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-17T19:25:22.523 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-17T19:25:22.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:22.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:25:22.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:22.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:25:22.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:25:22.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:22.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:25:22.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:22.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:25:22.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:22.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:25:22.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:22:26.938364+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:22.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:25:23.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:25:23.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:23.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:25:23.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:23.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:25:23.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:23.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:25:24.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:22:26.938364+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:24.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:25:25.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:25:25.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:25.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:25:25.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:25.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:25:25.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:25.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:25:25.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:25:23.435110+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:25.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:25:25.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-17T19:25:25.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-17T19:25:25.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-17T19:25:25.701 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-17T19:25:25.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-17T19:25:25.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-17T19:25:25.754 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-17T19:25:25.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=121 2024-09-17T19:25:25.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:25:25.782 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:25:25.782 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:25:25.782 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:25:25.782 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:25:25.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:25:25.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:25:25.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:25:25.823 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:25:25.823 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:25:25.823 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:25:25.824 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:25:25.825 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-17T19:25:25.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:25:25.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:25:25.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:25:25.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-17T19:25:25.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-17T19:25:25.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-17T19:25:25.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-17T19:25:25.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:25:25.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:25.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:25:25.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:25.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:25:26.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:26.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-17T19:25:26.468 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-17T19:25:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-17T19:22:26.938364+0000 last_deep_scrub_stamp 2024-09-17T19:25:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:25:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-17T19:25:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:25:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:25:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:25:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:25:26.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:22:26.938364+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:26.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:25:27.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:25:27.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:27.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:25:27.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:27.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:25:27.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:27.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:25:28.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:22:26.938364+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:28.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:25:29.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:25:29.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:29.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:25:29.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:29.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:25:29.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:29.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:25:29.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:22:26.938364+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:29.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:25:30.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:25:30.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:30.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:25:30.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:30.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:25:30.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:30.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:25:30.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:22:26.938364+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:30.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:25:31.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:25:31.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:25:31.778 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:25:31.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:25:31.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:25:31.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:25:31.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:25:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:25:27.658419+0000 '>' 2024-09-17T19:22:26.938364+0000 2024-09-17T19:25:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:25:32.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-17T19:25:32.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-17T19:25:32.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-17T19:25:32.164 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-17T19:25:32.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-17T19:25:32.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-17T19:25:32.222 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-17T19:25:32.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=121 2024-09-17T19:25:32.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-09-17T19:25:32.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:25:32.252 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:25:32.253 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:25:32.253 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:25:32.253 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T19:25:32.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:25:32.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:25:32.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:25:32.296 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:25:32.297 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:25:32.297 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:25:32.297 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-17T19:25:32.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:25:32.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:25:32.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:25:32.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:25:32.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-17T19:25:32.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-17T19:25:32.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-17T19:25:32.751 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' removed 2024-09-17T19:25:32.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T19:25:32.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:25:32.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:25:32.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:25:32.773 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:25:32.773 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:25:32.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:25:32.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:25:32.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:25:33.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:25:33.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:25:33.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:25:33.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:25:33.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:25:33.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:25:33.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:25:33.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:25:33.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:25:33.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:25:33.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:25:33.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:25:33.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:25:33.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:25:33.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:25:33.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:33.132 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:33.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:25:33.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:25:33.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:25:33.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:25:33.134 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:25:33.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:25:33.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:25:33.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:25:33.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:25:33.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:25:33.139 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:25:33.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:25:33.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:25:33.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:25:33.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:25:33.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:25:33.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:25:33.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:25:33.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:25:33.144 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:25:33.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:25:33.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:25:33.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:25:33.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:25:33.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:33.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:33.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:25:33.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:25:33.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:25:33.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T19:25:33.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T19:25:33.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:33.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:33.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T19:25:33.154 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T19:25:33.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T19:25:33.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T19:25:33.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T19:25:33.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T19:25:33.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-09-17T19:25:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T19:25:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:25:33.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:25:33.637 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:25:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:33.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:25:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-17T19:25:33.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T19:25:33.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T19:25:33.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:25:33.858 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:25:33.859 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T19:25:33.859 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T19:25:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:25:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:25:33.860 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:25:33.860 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:25:33.860 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:33.861 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:33.861 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:25:33.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:25:33.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T19:25:33.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T19:25:33.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:25:33.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:25:33.976 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T19:25:33.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T19:25:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:25:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:25:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:25:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:25:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:33.978 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:25:33.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:25:33.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T19:25:34.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:34.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:25:34.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:25:34.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:25:34.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:25:34.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:25:34.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:25:34.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:25:34.999 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:25:34.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:25:34.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e509dc6b-96d3-48d2-a145-04014a106f18' 2024-09-17T19:25:34.999 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:25:35.000 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:25:35.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCv1+lmCrj+ABAAl0G7DpjDP3rJ/Rf02nhpiw== 2024-09-17T19:25:35.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCv1+lmCrj+ABAAl0G7DpjDP3rJ/Rf02nhpiw=="}' 2024-09-17T19:25:35.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e509dc6b-96d3-48d2-a145-04014a106f18 -i td/osd-scrub-repair/0/new.json 2024-09-17T19:25:35.259 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:25:35.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T19:25:35.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCv1+lmCrj+ABAAl0G7DpjDP3rJ/Rf02nhpiw== --osd-uuid e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:25:35.317 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:35.316+0000 7f90592c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:35.319 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:35.316+0000 7f90592c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:35.320 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:35.320+0000 7f90592c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:35.321 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:35.320+0000 7f90592c07c0 -1 bdev(0x559bcbde5c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T19:25:35.321 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:35.320+0000 7f90592c07c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T19:25:41.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T19:25:41.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:25:41.690 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T19:25:41.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T19:25:41.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:25:42.104 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T19:25:42.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T19:25:42.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:25:42.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:25:42.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:25:42.105 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:25:42.130 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:42.128+0000 7f33477a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:42.133 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:42.132+0000 7f33477a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:42.135 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:42.132+0000 7f33477a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:42.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:25:42.452 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:25:42.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:42.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:42.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:43.158 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:43.155+0000 7f33477a77c0 -1 Falling back to public interface 2024-09-17T19:25:43.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:43.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:43.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:25:43.792 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:25:43.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:43.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:44.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:45.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:45.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:45.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:25:45.129 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:25:45.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:45.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:45.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:45.896 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:25:45.895+0000 7f33477a77c0 -1 osd.0 0 log_to_monitors true 2024-09-17T19:25:46.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:46.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:46.474 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:25:46.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:25:46.474 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:46.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:46.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:47.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:47.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:47.898 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:25:47.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:25:47.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:47.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:48.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:49.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:49.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:49.225 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:25:49.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:25:49.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:49.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:49.727 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:50.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:50.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:50.729 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:25:50.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:25:50.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:50.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:51.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:52.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:52.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:52.063 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:25:52.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:25:52.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:52.064 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:52.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:53.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:53.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:53.398 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:25:53.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:25:53.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:53.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:53.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:54.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:54.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:54.734 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:25:54.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:25:54.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:54.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:55.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:56.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:56.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:56.069 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:25:56.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:25:56.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:56.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:56.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:57.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:57.485 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:25:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:25:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:57.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:57.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:25:58.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:25:58.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:25:58.851 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:25:58.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:25:58.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:25:58.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:25:59.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:00.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:00.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:00.183 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:26:00.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:26:00.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:00.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:00.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:01.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:01.747 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:26:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:26:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:01.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:02.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:03.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:03.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:03.512 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:26:03.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:26:03.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:03.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:04.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:05.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:05.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:05.298 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:26:05.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:26:05.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:05.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:05.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:06.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:06.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:06.934 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:26:06.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:26:06.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:06.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:07.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:08.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:08.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:08.326 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:26:08.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:26:08.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:08.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:09.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:10.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:10.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:10.120 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:26:10.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:26:10.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:10.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:26:10.454 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3757576336,v1:127.0.0.1:6803/3757576336] [v2:127.0.0.1:6804/3757576336,v1:127.0.0.1:6805/3757576336] exists,up e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:26:10.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:26:10.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:26:10.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:26:10.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:26:10.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:26:10.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:26:10.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:26:10.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:26:10.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:26:10.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:26:10.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7e77352f-83a0-47c5-94fe-f2a308bf89c3' 2024-09-17T19:26:10.463 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:26:10.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:26:10.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDS1+lmkL/ZHBAAnJhzOVV8Lh/MyrLC8GbYeg== 2024-09-17T19:26:10.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDS1+lmkL/ZHBAAnJhzOVV8Lh/MyrLC8GbYeg=="}' 2024-09-17T19:26:10.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e77352f-83a0-47c5-94fe-f2a308bf89c3 -i td/osd-scrub-repair/1/new.json 2024-09-17T19:26:10.850 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:26:10.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T19:26:10.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDS1+lmkL/ZHBAAnJhzOVV8Lh/MyrLC8GbYeg== --osd-uuid 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:26:10.905 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:10.903+0000 7f72f4acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:10.969 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:10.967+0000 7f72f4acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:10.971 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:10.967+0000 7f72f4acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:10.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:10.971+0000 7f72f4acf7c0 -1 bdev(0x557814140800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T19:26:10.972 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:10.971+0000 7f72f4acf7c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T19:26:17.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T19:26:17.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:26:17.423 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T19:26:17.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T19:26:17.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:26:17.823 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T19:26:17.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T19:26:17.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:26:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:26:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:26:17.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:26:17.855 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:17.851+0000 7f39e10287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:17.857 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:17.855+0000 7f39e10287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:17.859 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:17.855+0000 7f39e10287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:26:18.181 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:26:18.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:18.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:18.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:19.367 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:19.363+0000 7f39e10287c0 -1 Falling back to public interface 2024-09-17T19:26:19.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:19.501 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:26:19.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:19.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:26:19.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:19.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:19.827 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:20.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:20.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:20.829 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:26:20.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:26:20.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:20.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:22.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:22.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:22.145 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:26:22.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:26:22.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:22.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:22.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:22.476 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:22.475+0000 7f39e10287c0 -1 osd.1 0 log_to_monitors true 2024-09-17T19:26:23.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:23.477 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:23.477 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:26:23.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:26:23.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:23.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:23.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:24.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:24.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:24.818 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:26:24.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:26:24.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:24.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:25.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:26.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:26.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:26.229 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:26:26.229 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:26:26.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:26.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:26.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:27.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:27.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:27.648 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:26:27.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:26:27.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:27.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:27.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:29.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:29.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:29.002 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:26:29.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:26:29.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:29.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:29.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:30.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:30.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:30.449 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:26:30.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:26:30.449 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:30.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:30.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:31.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:31.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:31.882 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:26:31.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:26:31.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:31.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:32.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:33.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:33.349 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:33.349 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:26:33.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:26:33.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:33.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:33.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:34.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:34.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:34.686 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:26:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:26:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:34.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:35.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:36.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:36.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:36.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:26:36.020 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:26:36.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:36.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:36.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:37.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:37.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:37.688 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:26:37.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:26:37.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:37.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:38.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:39.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:39.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:39.628 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:26:39.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:26:39.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:39.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:40.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:41.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:41.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:41.078 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:26:41.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:26:41.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:41.079 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:41.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:42.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:42.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:42.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:26:42.639 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:26:42.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:42.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:42.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:43.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:43.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:43.990 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:26:43.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:26:43.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:43.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:44.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:45.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:45.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:45.577 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:26:45.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:26:45.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:45.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:46.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:47.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:47.027 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:47.028 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:26:47.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:26:47.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:47.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:26:47.398 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/578150649,v1:127.0.0.1:6811/578150649] [v2:127.0.0.1:6812/578150649,v1:127.0.0.1:6813/578150649] exists,up 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:26:47.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:26:47.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:26:47.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:26:47.405 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:26:47.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:26:47.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e5fc78f5-ba7a-4f55-8984-61565fe9307c 2024-09-17T19:26:47.408 INFO:tasks.workunit.client.0.smithi105.stdout:add osd2 e5fc78f5-ba7a-4f55-8984-61565fe9307c 2024-09-17T19:26:47.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e5fc78f5-ba7a-4f55-8984-61565fe9307c' 2024-09-17T19:26:47.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:26:47.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD31+lmixp3GRAAB2Vp+L33N7AOwY8xQDj2/w== 2024-09-17T19:26:47.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD31+lmixp3GRAAB2Vp+L33N7AOwY8xQDj2/w=="}' 2024-09-17T19:26:47.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e5fc78f5-ba7a-4f55-8984-61565fe9307c -i td/osd-scrub-repair/2/new.json 2024-09-17T19:26:47.898 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:26:47.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2024-09-17T19:26:47.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD31+lmixp3GRAAB2Vp+L33N7AOwY8xQDj2/w== --osd-uuid e5fc78f5-ba7a-4f55-8984-61565fe9307c 2024-09-17T19:26:47.954 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:47.950+0000 7ff721e597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:47.982+0000 7ff721e597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:47.983 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:47.982+0000 7ff721e597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:47.984 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:47.982+0000 7ff721e597c0 -1 bdev(0x560cb8a65c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2024-09-17T19:26:47.984 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:47.982+0000 7ff721e597c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-09-17T19:26:54.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-09-17T19:26:54.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:26:54.182 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd2 key to auth repository 2024-09-17T19:26:54.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-17T19:26:54.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:26:54.582 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.2 2024-09-17T19:26:54.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-17T19:26:54.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:26:54.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:26:54.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:26:54.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:26:54.616 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:54.614+0000 7fdecea987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:54.645 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:54.642+0000 7fdecea987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:54.647 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:54.642+0000 7fdecea987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:26:54.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-17T19:26:54.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:26:54.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:26:54.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:26:54.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:26:54.926 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:26:54.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:54.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:26:54.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:54.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:26:55.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:55.950 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:55.946+0000 7fdecea987c0 -1 Falling back to public interface 2024-09-17T19:26:56.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:56.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:56.257 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:26:56.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:26:56.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:56.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:26:56.573 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:57.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:57.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:57.576 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:26:57.576 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:26:57.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:57.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:26:57.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:26:58.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:26:58.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:26:58.907 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:26:58.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:26:58.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:26:58.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:26:58.969 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:26:58.966+0000 7fdecea987c0 -1 osd.2 0 log_to_monitors true 2024-09-17T19:26:59.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:00.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:00.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:00.324 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:27:00.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:27:00.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:00.325 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:00.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:01.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:01.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:01.735 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:27:01.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:27:01.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:01.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:02.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:03.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:03.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:03.161 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:27:03.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:27:03.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:03.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:03.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:04.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:04.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:04.499 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:27:04.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:27:04.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:04.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:04.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:05.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:05.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:05.851 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:27:05.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:27:05.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:05.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:06.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:07.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:07.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:07.177 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:27:07.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:27:07.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:07.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:07.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:08.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:08.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:08.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:27:08.497 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:27:08.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:08.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:08.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:09.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:09.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:09.852 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:27:09.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:27:09.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:09.853 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:10.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:11.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:11.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:11.174 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:27:11.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:27:11.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:11.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:11.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:12.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:12.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:12.508 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:27:12.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:27:12.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:12.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:13.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:14.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:14.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:14.084 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:27:14.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:27:14.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:14.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:14.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:15.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:15.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:15.459 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:27:15.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:27:15.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:15.460 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:15.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:16.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:16.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:16.936 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:27:16.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:27:16.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:16.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:17.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:18.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:18.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:18.701 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:27:18.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:27:18.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:18.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:19.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:20.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:20.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:20.603 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:27:20.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:27:20.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:20.604 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:20.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:21.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:21.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:21.941 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:27:21.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:27:21.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:21.942 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:22.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:23.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:23.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:23.669 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:27:23.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:27:23.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:23.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:27:24.014 INFO:tasks.workunit.client.0.smithi105.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1827183102,v1:127.0.0.1:6819/1827183102] [v2:127.0.0.1:6820/1827183102,v1:127.0.0.1:6821/1827183102] exists,up e5fc78f5-ba7a-4f55-8984-61565fe9307c 2024-09-17T19:27:24.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:27:24.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:27:24.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:27:24.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-09-17T19:27:24.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T19:27:24.368 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T19:27:24.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T19:27:24.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T19:27:24.866 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T19:27:24.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:27:25.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T19:27:28.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-09-17T19:27:28.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-17T19:27:29.171 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'foo' created 2024-09-17T19:27:29.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:27:30.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-09-17T19:27:30.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=ecpool 2024-09-17T19:27:30.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-17T19:27:30.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-17T19:27:30.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-17T19:27:30.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-09-17T19:27:30.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-09-17T19:27:30.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-09-17T19:27:31.531 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' created 2024-09-17T19:27:31.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:27:32.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-17T19:27:32.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-09-17T19:27:33.028 INFO:tasks.workunit.client.0.smithi105.stderr:set pool 3 allow_ec_overwrites to true 2024-09-17T19:27:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-17T19:27:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:27:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:27:33.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:27:33.052 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:27:33.052 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:27:33.052 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:27:33.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:27:33.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:27:33.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:27:33.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:27:33.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:27:33.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:27:33.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:27:33.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:27:33.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:27:33.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:27:33.618 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:27:33.618 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:27:33.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:27:33.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:33.618 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:27:33.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-17T19:27:33.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-17T19:27:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-17T19:27:33.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:33.833 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:27:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-17T19:27:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-17T19:27:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672977' 2024-09-17T19:27:33.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:33.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:27:34.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-17T19:27:34.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-17T19:27:34.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672977 2-64424509451' 2024-09-17T19:27:34.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:34.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-17T19:27:34.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:34.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:27:34.136 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-17T19:27:34.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:34.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-17T19:27:34.138 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-17T19:27:34.138 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836503 2024-09-17T19:27:34.139 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:27:34.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-17T19:27:34.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:27:35.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:27:35.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:27:35.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836503 2024-09-17T19:27:35.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:27:36.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:27:36.812 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:27:37.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836503 2024-09-17T19:27:37.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:37.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-17T19:27:37.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:37.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:27:37.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-17T19:27:37.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:37.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-17T19:27:37.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-17T19:27:37.140 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672977 2024-09-17T19:27:37.140 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:27:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-17T19:27:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:37.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-17T19:27:37.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:37.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:27:37.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-17T19:27:37.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:37.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-17T19:27:37.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-17T19:27:37.476 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509451 2024-09-17T19:27:37.476 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:27:37.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-17T19:27:37.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:27:37.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:37.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:38.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:27:38.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:27:38.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:27:38.295 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:27:38.296 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:27:38.296 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:27:38.296 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:27:38.296 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:27:38.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:27:38.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:27:38.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:38.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:27:39.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:27:39.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:27:39.005 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:27:39.005 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:27:39.005 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:27:39.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:27:39.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:27:39.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:27:39.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:27:39.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:27:39.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:27:39.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:27:39.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:27:39.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:27:39.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:27:39.434 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:27:39.434 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:27:39.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:27:39.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:39.434 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:27:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-17T19:27:39.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-17T19:27:39.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-17T19:27:39.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:39.587 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:27:39.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-17T19:27:39.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-17T19:27:39.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672979' 2024-09-17T19:27:39.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:39.733 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:27:39.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509453 2024-09-17T19:27:39.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509453 2024-09-17T19:27:39.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672979 2-64424509453' 2024-09-17T19:27:39.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:39.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-17T19:27:39.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:39.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:27:39.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-17T19:27:39.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:39.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-17T19:27:39.874 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836505 2024-09-17T19:27:39.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-17T19:27:39.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:27:40.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-17T19:27:40.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:40.193 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-17T19:27:40.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:40.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:27:40.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-17T19:27:40.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:40.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-17T19:27:40.198 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672979 2024-09-17T19:27:40.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-17T19:27:40.199 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:27:40.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672978 -lt 42949672979 2024-09-17T19:27:40.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:27:41.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:27:41.537 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:27:41.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-17T19:27:41.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:41.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509453 2024-09-17T19:27:41.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:41.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:27:41.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509453 2024-09-17T19:27:41.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:41.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509453 2024-09-17T19:27:41.869 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.2 seq 64424509453 2024-09-17T19:27:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-17T19:27:41.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:27:42.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-17T19:27:42.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:27:42.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:42.186 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:42.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:27:42.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:27:42.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:27:42.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:27:42.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:27:42.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:27:42.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:27:42.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:27:42.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:27:42.909 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:27:42.909 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:42.909 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:43.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:27:43.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:27:43.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:27:43.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-09-17T19:27:43.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:27:43.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-09-17T19:27:43.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-09-17T19:27:43.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:27:43.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:27:43.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-09-17T19:27:43.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:27:43.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:27:43.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:27:43.747 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:27:43.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:27:44.224 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:27:44.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:27:44.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:27:44.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:27:44.490 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-09-17T19:27:44.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:27:44.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:27:44.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:27:44.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:27:44.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:27:44.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:27:45.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:27:47.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:27:47.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:27:47.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:27:47.945 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:27:47.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:27:47.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:27:47.945 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:27:47.946 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:27:47.946 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:27:47.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:27:47.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:27:47.948 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:27:47.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:27:47.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:27:47.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:27:47.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:27:47.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:27:47.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:27:47.982 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:27:47.981+0000 7f97221507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:27:47.985 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:27:47.985+0000 7f97221507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:27:47.987 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:27:47.985+0000 7f97221507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:27:48.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:27:48.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:27:48.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:27:48.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:27:48.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:27:48.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:48.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:27:48.319 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:27:48.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:48.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:27:48.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:49.014 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:27:49.013+0000 7f97221507c0 -1 Falling back to public interface 2024-09-17T19:27:49.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:49.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:49.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:27:49.677 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:27:49.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:49.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:27:50.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:27:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:27:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:51.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:27:51.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:27:51.872 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:27:51.873+0000 7f97221507c0 -1 osd.1 37 log_to_monitors true 2024-09-17T19:27:52.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:27:52.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:27:52.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:27:52.381 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:27:52.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:27:52.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:27:52.753 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 41 up_thru 31 down_at 38 last_clean_interval [10,37) [v2:127.0.0.1:6810/2683230504,v1:127.0.0.1:6811/2683230504] [v2:127.0.0.1:6812/2683230504,v1:127.0.0.1:6813/2683230504] exists,up 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:27:52.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:27:52.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:27:52.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:27:52.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:27:52.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:27:52.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:27:52.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:27:52.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:27:52.755 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:27:52.755 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:27:52.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:27:52.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:27:52.755 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:27:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:27:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:27:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:27:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:27:52.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:27:52.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:27:53.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:27:53.244 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:27:53.245 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:27:53.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:27:53.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:53.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:27:53.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836509 2024-09-17T19:27:53.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836509 2024-09-17T19:27:53.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509' 2024-09-17T19:27:53.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:53.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:27:53.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659138 2024-09-17T19:27:53.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659138 2024-09-17T19:27:53.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-176093659138' 2024-09-17T19:27:53.702 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:27:53.702 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:27:53.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509457 2024-09-17T19:27:53.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509457 2024-09-17T19:27:53.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836509 1-176093659138 2-64424509457' 2024-09-17T19:27:53.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:53.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836509 2024-09-17T19:27:53.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:53.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:27:53.979 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836509 2024-09-17T19:27:53.979 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836509 2024-09-17T19:27:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-17T19:27:53.981 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836509 2024-09-17T19:27:53.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:27:54.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-17T19:27:54.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:54.368 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659138 2024-09-17T19:27:54.368 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:54.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:27:54.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659138 2024-09-17T19:27:54.370 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:54.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659138 2024-09-17T19:27:54.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2024-09-17T19:27:54.372 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 176093659138 2024-09-17T19:27:54.372 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:27:54.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 176093659138 2024-09-17T19:27:54.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:27:55.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:27:55.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:27:56.019 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659138 -lt 176093659138 2024-09-17T19:27:56.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:27:56.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509457 2024-09-17T19:27:56.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:27:56.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:27:56.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509457 2024-09-17T19:27:56.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:27:56.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509457 2024-09-17T19:27:56.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-17T19:27:56.024 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509457 2024-09-17T19:27:56.024 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:27:56.353 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-17T19:27:56.354 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:27:56.354 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:56.355 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:56.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:27:56.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:27:56.786 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:27:56.786 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:27:56.786 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:27:56.786 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:27:56.787 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:27:56.787 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:27:57.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:27:57.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:27:57.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:57.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:57.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:27:57.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:27:57.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:27:57.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:27:57.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:27:57.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:27:57.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:27:57.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:27:57.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:27:57.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:27:57.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:27:57.621 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:27:57.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:27:57.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:27:57.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:27:57.960 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:27:57.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:57.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:58.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:27:58.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T19:27:58.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:27:58.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:27:58.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:27:58.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:27:58.328 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:27:58.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:27:58.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:27:58.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:27:58.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:27:58.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:27:58.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:27:58.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:27:58.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:27:58.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:27:58.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:27:58.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:27:58.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:27:58.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:27:58.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:27:58.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:27:59.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:27:59.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:27:59.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:27:59.261 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:27:59.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:27:59.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:28:00.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:28:00.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:28:00.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-17T19:28:00.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:28:00.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T19:28:00.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:28:00.487 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:00.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:00.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:00.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:00.488 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:00.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:00.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:00.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:28:00.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:00.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:00.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:28:01.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:28:01.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-09-17T19:28:01.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:28:01.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:28:01.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:28:01.635 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:28:01.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:28:02.204 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:28:02.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:28:02.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:28:02.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:28:02.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-09-17T19:28:02.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-17T19:28:02.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:28:02.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-09-17T19:28:02.497 INFO:tasks.workunit.client.0.smithi105.stderr:1+0 records in 2024-09-17T19:28:02.497 INFO:tasks.workunit.client.0.smithi105.stderr:1+0 records out 2024-09-17T19:28:02.497 INFO:tasks.workunit.client.0.smithi105.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 0.000161286 s, 12.7 MB/s 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:28:02.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:28:02.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:28:02.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:28:02.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:28:02.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:28:05.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:28:05.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:28:05.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:28:05.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:28:05.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:28:05.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:28:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:28:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:28:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:28:05.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:05.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:28:05.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:28:05.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:28:05.699 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:28:05.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:28:05.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:28:05.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:28:05.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:28:05.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:28:05.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:28:05.734 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:05.732+0000 7f53270867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:05.770 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:05.768+0000 7f53270867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:05.771 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:05.768+0000 7f53270867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:06.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:28:06.036 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:28:06.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:06.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:06.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:07.062 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:07.060+0000 7f53270867c0 -1 Falling back to public interface 2024-09-17T19:28:07.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:07.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:07.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:28:07.391 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:07.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:07.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:07.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:08.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:08.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:08.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:28:08.717 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:28:08.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:08.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:09.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:09.956 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:09.956+0000 7f53270867c0 -1 osd.0 46 log_to_monitors true 2024-09-17T19:28:10.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:10.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:10.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:28:10.054 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:28:10.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:10.054 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 51 up_thru 42 down_at 47 last_clean_interval [5,46) [v2:127.0.0.1:6802/1152273461,v1:127.0.0.1:6803/1152273461] [v2:127.0.0.1:6804/1152273461,v1:127.0.0.1:6805/1152273461] exists,up e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:28:10.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:28:10.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:28:10.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:28:10.493 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:28:10.493 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:28:10.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:28:10.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:28:10.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:28:10.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:28:10.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:28:10.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:28:10.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:28:10.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:28:10.599 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:28:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:28:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:28:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:28:10.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:10.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:28:11.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332098 2024-09-17T19:28:11.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332098 2024-09-17T19:28:11.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-219043332098' 2024-09-17T19:28:11.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:11.209 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:28:11.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659142 2024-09-17T19:28:11.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659142 2024-09-17T19:28:11.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-219043332098 1-176093659142' 2024-09-17T19:28:11.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:11.457 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:28:11.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-17T19:28:11.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-17T19:28:11.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-219043332098 1-176093659142 2-64424509461' 2024-09-17T19:28:11.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:11.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-219043332098 2024-09-17T19:28:11.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:11.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:28:11.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-219043332098 2024-09-17T19:28:11.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:11.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332098 2024-09-17T19:28:11.675 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 219043332098' 2024-09-17T19:28:11.675 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 219043332098 2024-09-17T19:28:11.676 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:12.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 219043332098 2024-09-17T19:28:12.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:28:13.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:28:13.044 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:13.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332098 -lt 219043332098 2024-09-17T19:28:13.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:13.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:13.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659142 2024-09-17T19:28:13.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:28:13.371 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659142 2024-09-17T19:28:13.371 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:13.372 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659142 2024-09-17T19:28:13.373 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659142' 2024-09-17T19:28:13.373 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 176093659142 2024-09-17T19:28:13.373 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:28:13.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659142 -lt 176093659142 2024-09-17T19:28:13.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:13.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-17T19:28:13.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:13.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:28:13.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-17T19:28:13.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:13.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-17T19:28:13.700 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-17T19:28:13.700 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509461 2024-09-17T19:28:13.700 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:28:14.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509461 2024-09-17T19:28:14.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:28:14.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:14.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:14.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:28:14.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:14.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:14.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:14.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:14.471 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:14.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:14.472 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:14.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:28:14.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:14.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:14.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:15.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:28:15.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-17T19:28:15.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:28:15.232 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-17T19:28:15.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:28:15.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:28:15.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:28:15.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:15.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:15.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:15.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:15.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:15.335 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:15.335 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:15.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:28:15.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:15.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:15.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:28:16.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:28:16.022 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:28:16.022 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:28:16.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:28:16.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:28:16.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:28:16.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:28:16.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:28:16.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:28:16.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:16.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:16.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:16.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:16.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:16.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:16.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:16.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:28:16.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:16.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:16.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:17.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:28:17.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:28:17.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:28:17.835 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:28:17.852 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:28:18.295 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:28:18.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:28:18.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:28:18.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:28:18.610 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-09-17T19:28:18.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:28:18.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:28:18.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:28:18.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:28:18.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:28:18.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:28:18.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:28:19.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:28:19.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-09-17T19:28:21.334 INFO:tasks.workunit.client.0.smithi105.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-09-17T19:28:21.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:28:21.883 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:28:21.884 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:28:21.886 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:28:21.886 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:28:21.886 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:28:21.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:21.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:28:21.887 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:28:21.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:28:21.888 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:28:21.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:28:21.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:28:21.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:28:21.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:28:21.922 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:21.920+0000 7ff01c6437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:21.925 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:21.924+0000 7ff01c6437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:21.926 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:21.924+0000 7ff01c6437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:22.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:28:22.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:28:22.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:28:22.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:28:22.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:28:22.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:22.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:28:22.223 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:28:22.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:22.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:28:22.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:23.443 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:23.440+0000 7ff01c6437c0 -1 Falling back to public interface 2024-09-17T19:28:23.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:23.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:23.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:28:23.551 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:23.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:23.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:28:23.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:24.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:24.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:24.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:28:24.866 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:28:24.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:24.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:28:25.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:26.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:28:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:28:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:26.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:28:26.283 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:26.280+0000 7ff01c6437c0 -1 osd.1 55 log_to_monitors true 2024-09-17T19:28:26.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:27.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:27.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:27.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:28:27.564 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:28:27.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:27.564 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:28:28.041 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [41,55) [v2:127.0.0.1:6810/344550774,v1:127.0.0.1:6811/344550774] [v2:127.0.0.1:6812/344550774,v1:127.0.0.1:6813/344550774] exists,up 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:28:28.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:28:28.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:28:28.042 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:28:28.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:28:28.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:28:28.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:28:28.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:28:28.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:28:28.158 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:28:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:28:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:28.553 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:28:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:28:28.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:28.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:28:28.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=219043332102 2024-09-17T19:28:28.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 219043332102 2024-09-17T19:28:28.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-219043332102' 2024-09-17T19:28:28.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:28.705 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:28:28.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070466 2024-09-17T19:28:28.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070466 2024-09-17T19:28:28.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-219043332102 1-253403070466' 2024-09-17T19:28:28.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:28.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:28:28.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509466 2024-09-17T19:28:28.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509466 2024-09-17T19:28:28.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-219043332102 1-253403070466 2-64424509466' 2024-09-17T19:28:28.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:28.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-219043332102 2024-09-17T19:28:28.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:28.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:28:28.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-219043332102 2024-09-17T19:28:28.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:28.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=219043332102 2024-09-17T19:28:28.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 219043332102' 2024-09-17T19:28:28.992 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 219043332102 2024-09-17T19:28:28.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:29.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332101 -lt 219043332102 2024-09-17T19:28:29.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:28:30.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:28:30.314 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:30.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 219043332102 -lt 219043332102 2024-09-17T19:28:30.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:30.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-253403070466 2024-09-17T19:28:30.638 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:30.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:28:30.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-253403070466 2024-09-17T19:28:30.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070466 2024-09-17T19:28:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 253403070466' 2024-09-17T19:28:30.641 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 253403070466 2024-09-17T19:28:30.642 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:28:30.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070466 -lt 253403070466 2024-09-17T19:28:30.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:30.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509466 2024-09-17T19:28:30.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:30.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:28:30.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509466 2024-09-17T19:28:30.985 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:30.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509466 2024-09-17T19:28:30.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-17T19:28:30.986 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509466 2024-09-17T19:28:30.987 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:28:31.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-17T19:28:31.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:28:31.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:31.329 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:31.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:28:31.718 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:31.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:31.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:31.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:31.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:31.719 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:31.720 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:32.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:28:32.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:32.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:32.032 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:32.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:28:32.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:28:32.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:28:32.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:28:32.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-09-17T19:28:32.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-09-17T19:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-09-17T19:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:28:32.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:28:32.865 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:28:33.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:28:33.417 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:28:33.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:28:33.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:28:33.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:28:33.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-09-17T19:28:33.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-17T19:28:33.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:28:33.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-09-17T19:28:33.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-09-17T19:28:34.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-09-17T19:28:34.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:28:34.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:28:34.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:28:34.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:28:34.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:28:34.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:28:34.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:28:34.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:28:34.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:28:34.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:28:34.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:28:34.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:28:34.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:28:34.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:28:34.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:28:34.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:28:34.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:28:34.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-09-17T19:28:36.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:28:36.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:28:36.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:28:36.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:28:36.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:28:36.933 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:28:36.934 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:28:36.934 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:28:36.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:28:36.936 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:28:36.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:28:36.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:28:36.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:28:36.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:28:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:28:36.938 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:28:36.970 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:36.968+0000 7f359b2fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:36.973 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:36.972+0000 7f359b2fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:36.975 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:36.972+0000 7f359b2fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:28:37.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:37.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:37.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:38.510 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:38.508+0000 7f359b2fe7c0 -1 Falling back to public interface 2024-09-17T19:28:38.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:28:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:38.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:39.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:40.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:40.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:40.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:28:40.181 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:28:40.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:40.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:40.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:41.145 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:41.143+0000 7f359b2fe7c0 -1 osd.0 63 log_to_monitors true 2024-09-17T19:28:41.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:41.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:41.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:28:41.517 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:28:41.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:41.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 68 up_thru 59 down_at 64 last_clean_interval [51,63) [v2:127.0.0.1:6802/2538697786,v1:127.0.0.1:6803/2538697786] [v2:127.0.0.1:6804/2538697786,v1:127.0.0.1:6805/2538697786] exists,up e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:28:41.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:28:41.870 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:28:41.870 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:28:41.871 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:28:41.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:28:41.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:28:41.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:28:41.976 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:28:41.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:28:41.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:28:41.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:28:41.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:28:41.977 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:28:42.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:28:42.420 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:42.420 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:28:42.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:28:42.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:42.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:28:42.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776130 2024-09-17T19:28:42.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776130 2024-09-17T19:28:42.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776130' 2024-09-17T19:28:42.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:42.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:28:42.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070470 2024-09-17T19:28:42.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070470 2024-09-17T19:28:42.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776130 1-253403070470' 2024-09-17T19:28:42.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:42.805 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:28:42.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509469 2024-09-17T19:28:42.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509469 2024-09-17T19:28:42.992 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776130 1-253403070470 2-64424509469' 2024-09-17T19:28:42.993 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:42.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-292057776130 2024-09-17T19:28:42.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:42.994 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:28:42.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-292057776130 2024-09-17T19:28:42.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:42.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776130 2024-09-17T19:28:42.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 292057776130' 2024-09-17T19:28:42.996 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 292057776130 2024-09-17T19:28:42.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:43.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 292057776130 2024-09-17T19:28:43.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:28:44.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:28:44.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:44.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776130 -lt 292057776130 2024-09-17T19:28:44.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:44.708 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-253403070470 2024-09-17T19:28:44.708 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:44.709 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:28:44.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-253403070470 2024-09-17T19:28:44.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:44.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070470 2024-09-17T19:28:44.712 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 253403070470' 2024-09-17T19:28:44.712 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 253403070470 2024-09-17T19:28:44.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:28:45.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070470 -lt 253403070470 2024-09-17T19:28:45.028 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:45.028 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509469 2024-09-17T19:28:45.028 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:45.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:28:45.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509469 2024-09-17T19:28:45.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509469 2024-09-17T19:28:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-17T19:28:45.033 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 64424509469 2024-09-17T19:28:45.033 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:28:45.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509470 -lt 64424509469 2024-09-17T19:28:45.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:28:45.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:45.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:45.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:28:45.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:45.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:45.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:45.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:45.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:45.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:45.784 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:46.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-17T19:28:46.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:46.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:46.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:46.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-17T19:28:46.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' -1 2024-09-17T19:28:46.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:28:46.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-17T19:28:46.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:28:46.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:28:46.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:28:46.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:46.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:46.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:46.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:46.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:46.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:46.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:46.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-17T19:28:46.910 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:46.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:46.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:47.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 6 2024-09-17T19:28:47.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-17T19:28:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:28:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:28:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:28:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:28:47.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:28:47.276 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:28:47.276 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:28:47.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:28:47.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:28:47.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:28:47.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:28:47.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:28:47.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:28:47.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:28:47.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:28:47.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:28:47.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:28:47.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:28:47.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:28:47.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:28:48.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:28:48.182 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:28:48.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:48.183 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:28:48.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:28:48.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:28:48.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:28:48.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:28:48.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:28:48.557 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:28:48.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:28:48.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:28:48.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:28:48.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:28:48.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:28:48.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:28:48.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:28:48.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T19:28:48.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:28:48.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:28:48.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-09-17T19:28:51.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T19:28:51.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:28:51.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:28:51.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T19:28:51.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:28:51.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:28:51.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:28:51.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:28:51.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:28:51.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T19:28:51.495 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.2 2024-09-17T19:28:51.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:28:51.495 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T19:28:51.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T19:28:51.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:28:51.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:28:51.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:28:51.526 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:51.523+0000 7fd8376db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:51.529 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:51.527+0000 7fd8376db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:51.530 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:51.527+0000 7fd8376db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:28:51.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:28:51.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:51.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:28:52.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:52.803 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:52.799+0000 7fd8376db7c0 -1 Falling back to public interface 2024-09-17T19:28:53.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:53.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:53.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:28:53.147 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:53.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:53.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:28:53.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:54.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:54.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:54.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:28:54.484 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:28:54.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:54.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:28:54.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:28:55.538 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:28:55.535+0000 7fd8376db7c0 -1 osd.2 70 log_to_monitors true 2024-09-17T19:28:55.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:28:55.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:28:55.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:28:55.802 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:28:55.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:28:55.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:28:56.145 INFO:tasks.workunit.client.0.smithi105.stderr:osd.2 up in weight 1 up_from 75 up_thru 68 down_at 71 last_clean_interval [15,70) [v2:127.0.0.1:6818/1122044879,v1:127.0.0.1:6819/1122044879] [v2:127.0.0.1:6820/1122044879,v1:127.0.0.1:6821/1122044879] exists,up e5fc78f5-ba7a-4f55-8984-61565fe9307c 2024-09-17T19:28:56.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:28:56.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:28:56.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:28:56.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:28:56.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:28:56.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:28:56.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:28:56.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:28:56.146 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:28:56.146 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:28:56.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:28:56.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:28:56.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:28:56.262 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:28:56.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:28:56.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:28:56.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:28:56.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:28:56.263 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:28:56.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:28:56.730 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:28:56.730 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:28:56.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:28:56.730 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:56.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:28:56.997 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776134 2024-09-17T19:28:56.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776134 2024-09-17T19:28:56.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776134' 2024-09-17T19:28:56.998 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:56.998 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:28:57.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070474 2024-09-17T19:28:57.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070474 2024-09-17T19:28:57.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776134 1-253403070474' 2024-09-17T19:28:57.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:28:57.158 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:28:57.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=322122547202 2024-09-17T19:28:57.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 322122547202 2024-09-17T19:28:57.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776134 1-253403070474 2-322122547202' 2024-09-17T19:28:57.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:57.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-292057776134 2024-09-17T19:28:57.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:57.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:28:57.418 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-292057776134 2024-09-17T19:28:57.418 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:57.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776134 2024-09-17T19:28:57.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 292057776134' 2024-09-17T19:28:57.419 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 292057776134 2024-09-17T19:28:57.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:57.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776132 -lt 292057776134 2024-09-17T19:28:57.745 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:28:58.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:28:58.747 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:28:59.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776134 -lt 292057776134 2024-09-17T19:28:59.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:59.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:59.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-253403070474 2024-09-17T19:28:59.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:28:59.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-253403070474 2024-09-17T19:28:59.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:59.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070474 2024-09-17T19:28:59.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 253403070474' 2024-09-17T19:28:59.087 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 253403070474 2024-09-17T19:28:59.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:28:59.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070474 -lt 253403070474 2024-09-17T19:28:59.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:28:59.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:28:59.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-322122547202 2024-09-17T19:28:59.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:28:59.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-322122547202 2024-09-17T19:28:59.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:28:59.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=322122547202 2024-09-17T19:28:59.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-09-17T19:28:59.444 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 322122547202 2024-09-17T19:28:59.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:28:59.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 322122547202 -lt 322122547202 2024-09-17T19:28:59.782 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:28:59.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:28:59.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:00.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:00.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:00.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:29:00.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:00.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:00.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:00.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:29:00.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:29:00.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:29:00.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:29:00.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:29:00.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:29:00.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:29:00.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:00.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:00.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:00.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:00.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:00.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:00.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:01.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:29:01.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:01.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:01.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:01.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:29:01.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:29:01.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:29:01.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:29:01.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:29:01.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:29:01.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:29:01.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:29:01.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:29:02.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:29:02.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:29:02.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:29:02.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:29:02.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:29:02.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:29:02.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:02.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:02.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:02.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:02.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:02.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:02.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:02.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:29:02.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:02.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:02.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:03.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:29:03.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:29:03.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:29:03.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:29:03.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:29:03.032 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:29:03.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:29:03.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:29:03.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:29:03.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:29:03.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-17T19:29:03.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:29:03.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:29:03.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-09-17T19:29:05.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-09-17T19:29:05.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:29:05.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:29:05.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:29:05.842 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:29:05.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:29:05.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:29:05.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:29:05.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-09-17T19:29:05.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-17T19:29:05.845 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.2 2024-09-17T19:29:05.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:29:05.845 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-09-17T19:29:05.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-17T19:29:05.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:29:05.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:29:05.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:29:05.875 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:05.871+0000 7fb1194317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:05.879 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:05.875+0000 7fb1194317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:05.880 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:05.879+0000 7fb1194317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:06.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:29:06.213 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:06.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:07.375 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:07.371+0000 7fb1194317c0 -1 Falling back to public interface 2024-09-17T19:29:07.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:07.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:07.539 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:29:07.539 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:07.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:07.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:29:07.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:08.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:08.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:08.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:29:08.876 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:29:08.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:08.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:29:09.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:10.183 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:10.179+0000 7fb1194317c0 -1 osd.2 77 log_to_monitors true 2024-09-17T19:29:10.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:10.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:10.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:29:10.203 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:29:10.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:10.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:29:10.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:11.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:11.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:11.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:29:11.537 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:29:11.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:11.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-17T19:29:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [75,77) [v2:127.0.0.1:6818/2890755148,v1:127.0.0.1:6819/2890755148] [v2:127.0.0.1:6820/2890755148,v1:127.0.0.1:6821/2890755148] exists,up e5fc78f5-ba7a-4f55-8984-61565fe9307c 2024-09-17T19:29:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:29:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:29:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:29:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:29:11.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:29:11.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:29:11.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:29:11.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:29:11.899 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:29:11.899 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:29:11.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:29:11.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:29:11.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:29:12.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:29:12.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:29:12.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:29:12.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:29:12.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:29:12.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:29:12.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:29:12.357 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:12.357 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:29:12.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:29:12.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:12.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:29:12.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776138 2024-09-17T19:29:12.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776138 2024-09-17T19:29:12.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776138' 2024-09-17T19:29:12.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:12.523 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:29:12.654 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=253403070478 2024-09-17T19:29:12.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 253403070478 2024-09-17T19:29:12.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776138 1-253403070478' 2024-09-17T19:29:12.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:12.655 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:29:12.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318274 2024-09-17T19:29:12.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318274 2024-09-17T19:29:12.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776138 1-253403070478 2-352187318274' 2024-09-17T19:29:12.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:12.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-292057776138 2024-09-17T19:29:12.794 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:12.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:29:12.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-292057776138 2024-09-17T19:29:12.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:12.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776138 2024-09-17T19:29:12.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 292057776138' 2024-09-17T19:29:12.798 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 292057776138 2024-09-17T19:29:12.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:13.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776137 -lt 292057776138 2024-09-17T19:29:13.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:29:14.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:29:14.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:14.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776138 -lt 292057776138 2024-09-17T19:29:14.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:14.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-253403070478 2024-09-17T19:29:14.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:29:14.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:14.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-253403070478 2024-09-17T19:29:14.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=253403070478 2024-09-17T19:29:14.467 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 253403070478' 2024-09-17T19:29:14.467 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 253403070478 2024-09-17T19:29:14.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:29:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 253403070478 -lt 253403070478 2024-09-17T19:29:14.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:14.809 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:14.809 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318274 2024-09-17T19:29:14.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:29:14.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318274 2024-09-17T19:29:14.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:14.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318274 2024-09-17T19:29:14.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-09-17T19:29:14.813 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 352187318274 2024-09-17T19:29:14.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:29:15.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318274 -lt 352187318274 2024-09-17T19:29:15.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:29:15.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:15.143 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:15.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:29:15.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:15.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:15.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:15.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:15.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:15.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:15.552 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:15.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:29:15.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:15.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:15.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:16.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:29:16.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:29:16.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:29:16.254 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-17T19:29:16.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:29:16.256 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:29:16.695 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:29:16.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:29:17.196 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:29:17.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:29:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:29:17.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:29:17.480 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-09-17T19:29:17.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:29:17.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:29:17.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-09-17T19:29:17.484 INFO:tasks.workunit.client.0.smithi105.stderr:2+0 records in 2024-09-17T19:29:17.484 INFO:tasks.workunit.client.0.smithi105.stderr:2+0 records out 2024-09-17T19:29:17.484 INFO:tasks.workunit.client.0.smithi105.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00013688 s, 29.9 MB/s 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:29:17.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:29:17.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:29:18.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:29:21.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:29:21.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:29:21.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:29:21.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:29:21.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:29:21.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:29:21.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:29:21.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:29:21.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:29:21.066 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:29:21.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:29:21.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:29:21.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:29:21.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:29:21.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:29:21.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:29:21.101 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:21.098+0000 7f9dcdcd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:21.103 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:21.098+0000 7f9dcdcd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:21.105 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:21.102+0000 7f9dcdcd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:21.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:29:21.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:29:21.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:29:21.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:29:21.390 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:29:21.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:21.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:29:21.391 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:29:21.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:21.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:21.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:22.167 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:22.162+0000 7f9dcdcd77c0 -1 Falling back to public interface 2024-09-17T19:29:22.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:22.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:22.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:29:22.746 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:22.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:22.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:23.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:24.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:24.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:24.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:29:24.076 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:29:24.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:24.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:24.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:25.161 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:25.158+0000 7f9dcdcd77c0 -1 osd.1 86 log_to_monitors true 2024-09-17T19:29:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:29:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:29:25.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:25.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:25.897 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 91 up_thru 82 down_at 87 last_clean_interval [59,86) [v2:127.0.0.1:6810/3642953426,v1:127.0.0.1:6811/3642953426] [v2:127.0.0.1:6812/3642953426,v1:127.0.0.1:6813/3642953426] exists,up 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:29:25.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:29:25.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:29:25.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:29:25.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:29:25.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:29:25.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:29:25.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:29:25.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:29:25.899 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:29:25.899 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:29:25.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:29:25.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:29:25.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:29:26.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:29:26.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:29:26.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:29:26.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:29:26.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:29:26.013 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:29:26.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:29:26.525 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:26.525 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:29:26.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:29:26.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:26.526 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:29:26.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776142 2024-09-17T19:29:26.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776142 2024-09-17T19:29:26.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776142' 2024-09-17T19:29:26.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:26.679 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:29:26.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023938 2024-09-17T19:29:26.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023938 2024-09-17T19:29:26.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776142 1-390842023938' 2024-09-17T19:29:26.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:26.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:29:27.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318278 2024-09-17T19:29:27.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318278 2024-09-17T19:29:27.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-292057776142 1-390842023938 2-352187318278' 2024-09-17T19:29:27.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:27.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-292057776142 2024-09-17T19:29:27.036 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:27.037 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:29:27.037 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-292057776142 2024-09-17T19:29:27.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:27.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776142 2024-09-17T19:29:27.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 292057776142' 2024-09-17T19:29:27.039 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 292057776142 2024-09-17T19:29:27.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:27.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776140 -lt 292057776142 2024-09-17T19:29:27.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:29:28.368 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:29:28.369 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:28.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776142 -lt 292057776142 2024-09-17T19:29:28.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:28.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023938 2024-09-17T19:29:28.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:28.713 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:29:28.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023938 2024-09-17T19:29:28.714 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:28.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023938 2024-09-17T19:29:28.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023938' 2024-09-17T19:29:28.716 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 390842023938 2024-09-17T19:29:28.716 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:29:29.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023938 -lt 390842023938 2024-09-17T19:29:29.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:29.085 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318278 2024-09-17T19:29:29.085 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:29.086 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:29:29.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318278 2024-09-17T19:29:29.087 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318278 2024-09-17T19:29:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318278' 2024-09-17T19:29:29.089 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 352187318278 2024-09-17T19:29:29.089 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:29:29.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318278 -lt 352187318278 2024-09-17T19:29:29.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:29:29.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:29.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:29.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:29:29.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:29.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:29.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:29.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:29.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:29.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:29.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:30.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:29:30.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:30.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:30.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:30.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:29:30.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T19:29:30.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:29:30.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T19:29:30.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:29:30.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:29:30.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:29:30.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:30.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:30.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:30.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:30.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:30.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:30.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:30.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:29:30.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:30.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:30.993 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:29:31.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:29:31.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:29:31.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:29:31.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:29:31.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:29:31.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:29:31.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:29:31.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:29:31.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:31.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:31.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:31.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:31.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:31.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:31.968 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:32.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:29:32.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:32.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:32.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:32.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:29:32.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:29:33.102 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:29:33.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:29:33.638 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:29:33.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:29:33.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:29:33.659 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:29:33.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:29:33.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:29:33.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:29:33.860 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:29:33.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:29:33.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:29:33.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:29:35.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:29:35.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:29:35.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:29:35.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:29:35.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:29:35.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:29:35.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:29:35.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-09-17T19:29:37.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:29:37.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:29:37.303 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:29:37.304 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:29:37.306 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:29:37.306 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:29:37.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:29:37.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:29:37.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:29:37.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:29:37.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:29:37.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:29:37.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:29:37.341 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:37.338+0000 7f8cdf8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:37.344 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:37.342+0000 7f8cdf8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:37.346 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:37.342+0000 7f8cdf8807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:37.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:29:37.634 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:29:37.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:37.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:29:37.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:38.631 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:38.626+0000 7f8cdf8807c0 -1 Falling back to public interface 2024-09-17T19:29:38.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:38.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:38.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:29:38.977 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:38.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:38.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:29:39.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:40.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:40.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:40.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:29:40.310 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:29:40.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:40.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:29:40.633 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:41.335 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:41.330+0000 7f8cdf8807c0 -1 osd.0 95 log_to_monitors true 2024-09-17T19:29:41.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:41.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:41.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:29:41.636 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:29:41.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:41.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 100 up_thru 91 down_at 96 last_clean_interval [68,95) [v2:127.0.0.1:6802/2424959314,v1:127.0.0.1:6803/2424959314] [v2:127.0.0.1:6804/2424959314,v1:127.0.0.1:6805/2424959314] exists,up e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:29:42.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:29:42.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:29:42.129 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:29:42.129 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:29:42.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:29:42.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:29:42.129 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:29:42.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:29:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:29:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:29:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:29:42.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:29:42.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:29:42.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:29:42.641 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:42.641 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:29:42.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:29:42.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:42.641 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:29:42.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729602 2024-09-17T19:29:42.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729602 2024-09-17T19:29:42.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602' 2024-09-17T19:29:42.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:42.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:29:43.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023942 2024-09-17T19:29:43.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023942 2024-09-17T19:29:43.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023942' 2024-09-17T19:29:43.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:43.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:29:43.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318282 2024-09-17T19:29:43.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318282 2024-09-17T19:29:43.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729602 1-390842023942 2-352187318282' 2024-09-17T19:29:43.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:43.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:43.275 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729602 2024-09-17T19:29:43.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:29:43.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729602 2024-09-17T19:29:43.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:43.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729602 2024-09-17T19:29:43.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729602' 2024-09-17T19:29:43.280 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 429496729602 2024-09-17T19:29:43.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:43.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 429496729602 2024-09-17T19:29:43.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:29:44.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:29:44.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:44.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729602 -lt 429496729602 2024-09-17T19:29:44.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:44.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:44.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023942 2024-09-17T19:29:44.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:29:44.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023942 2024-09-17T19:29:44.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:44.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023942 2024-09-17T19:29:44.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023942' 2024-09-17T19:29:44.953 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 390842023942 2024-09-17T19:29:44.954 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:29:45.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023942 -lt 390842023942 2024-09-17T19:29:45.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:45.314 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318282 2024-09-17T19:29:45.314 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:45.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:29:45.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318282 2024-09-17T19:29:45.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:45.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318282 2024-09-17T19:29:45.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318282' 2024-09-17T19:29:45.318 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 352187318282 2024-09-17T19:29:45.318 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:29:45.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318282 -lt 352187318282 2024-09-17T19:29:45.655 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:29:45.656 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:45.656 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:46.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:29:46.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:46.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:46.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:46.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:46.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:46.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:46.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:46.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:29:46.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:46.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:46.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:46.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:29:46.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-17T19:29:46.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:29:46.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-17T19:29:46.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:29:46.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:29:46.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:29:46.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:46.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:46.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:46.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:46.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:46.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:46.898 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:47.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:29:47.201 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:47.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:47.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:47.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:29:47.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-17T19:29:47.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:29:47.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:29:47.568 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:29:47.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:29:47.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:29:47.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:29:47.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:29:47.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:29:47.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:29:47.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:29:47.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:29:47.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:29:48.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:29:48.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:29:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:29:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:29:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:29:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:29:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:29:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:29:48.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:29:48.466 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:29:48.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:29:48.467 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:29:48.856 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:29:48.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:29:48.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:29:48.858 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:29:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:29:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:29:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:29:49.165 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:29:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:29:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:29:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:29:49.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-09-17T19:29:51.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:29:51.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:29:51.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:29:51.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:29:51.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:29:51.407 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:29:51.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:29:51.408 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:29:51.408 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:29:51.408 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:29:51.408 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:51.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:29:51.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:29:51.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:29:51.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:29:51.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:29:51.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:29:51.411 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:29:51.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:29:51.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:29:51.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:29:51.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:29:51.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:29:51.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:29:51.445 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:51.442+0000 7fe534cfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:51.447 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:51.442+0000 7fe534cfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:51.449 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:51.446+0000 7fe534cfd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:29:51.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:51.743 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:52.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:52.999 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:52.994+0000 7fe534cfd7c0 -1 Falling back to public interface 2024-09-17T19:29:53.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:53.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:53.084 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:29:53.084 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:53.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:53.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:53.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:54.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:29:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:29:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:54.407 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:54.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:55.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:55.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:55.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:29:55.739 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:29:55.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:55.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:55.766 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:29:55.766+0000 7fe534cfd7c0 -1 osd.1 102 log_to_monitors true 2024-09-17T19:29:56.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:29:57.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:29:57.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:29:57.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:29:57.129 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:29:57.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:29:57.130 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:29:57.643 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [91,102) [v2:127.0.0.1:6810/2836747751,v1:127.0.0.1:6811/2836747751] [v2:127.0.0.1:6812/2836747751,v1:127.0.0.1:6813/2836747751] exists,up 7e77352f-83a0-47c5-94fe-f2a308bf89c3 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:29:57.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:29:57.646 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:29:57.646 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:29:57.647 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:29:57.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:29:57.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:29:57.647 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:29:57.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:29:57.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:29:57.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:29:57.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:29:57.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:29:57.761 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:29:58.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:29:58.077 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:29:58.077 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:29:58.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:29:58.077 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:58.077 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:29:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729606 2024-09-17T19:29:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729606 2024-09-17T19:29:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729606' 2024-09-17T19:29:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:58.227 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:29:58.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500674 2024-09-17T19:29:58.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500674 2024-09-17T19:29:58.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729606 1-459561500674' 2024-09-17T19:29:58.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:29:58.366 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:29:58.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318286 2024-09-17T19:29:58.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318286 2024-09-17T19:29:58.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-429496729606 1-459561500674 2-352187318286' 2024-09-17T19:29:58.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:29:58.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-429496729606 2024-09-17T19:29:58.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:29:58.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:29:58.501 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-429496729606 2024-09-17T19:29:58.501 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:29:58.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729606 2024-09-17T19:29:58.502 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 429496729606' 2024-09-17T19:29:58.502 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 429496729606 2024-09-17T19:29:58.502 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:29:58.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729605 -lt 429496729606 2024-09-17T19:29:58.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:29:59.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:29:59.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:30:00.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729606 -lt 429496729606 2024-09-17T19:30:00.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:00.156 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500674 2024-09-17T19:30:00.157 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:00.159 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:30:00.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500674 2024-09-17T19:30:00.160 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:00.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500674 2024-09-17T19:30:00.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500674' 2024-09-17T19:30:00.161 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 459561500674 2024-09-17T19:30:00.162 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:30:00.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500674 -lt 459561500674 2024-09-17T19:30:00.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:00.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318286 2024-09-17T19:30:00.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:00.488 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:30:00.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318286 2024-09-17T19:30:00.489 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:00.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318286 2024-09-17T19:30:00.491 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318286' 2024-09-17T19:30:00.491 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 352187318286 2024-09-17T19:30:00.491 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:30:00.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318286 -lt 352187318286 2024-09-17T19:30:00.816 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:30:00.816 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:00.816 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:01.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:30:01.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:30:01.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:30:01.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:30:01.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:30:01.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:30:01.211 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:30:01.212 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:30:01.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:30:01.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:30:01.529 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:01.530 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:30:01.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:30:02.422 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:30:02.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:30:02.861 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:30:02.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:30:02.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:30:02.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:30:03.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-09-17T19:30:03.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-09-17T19:30:03.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-09-17T19:30:03.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-17T19:30:03.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-09-17T19:30:03.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-09-17T19:30:03.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-17T19:30:03.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:30:03.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:30:03.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:30:03.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:30:03.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:30:03.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:30:04.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:30:04.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-09-17T19:30:04.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:30:04.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:30:04.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:30:04.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:30:04.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:30:04.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-09-17T19:30:06.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:30:06.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:30:06.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:30:06.559 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:30:06.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:30:06.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:30:06.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:30:06.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:30:06.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:30:06.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:30:06.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:30:06.562 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:30:06.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:30:06.563 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:30:06.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:30:06.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:30:06.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:30:06.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:30:06.594 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:06.593+0000 7f9835d757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:06.596 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:06.597+0000 7f9835d757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:06.598 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:06.597+0000 7f9835d757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:06.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:30:06.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:30:06.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:30:06.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:30:06.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:30:06.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:06.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:30:06.892 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:30:06.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:06.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:07.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:07.879 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:07.877+0000 7f9835d757c0 -1 Falling back to public interface 2024-09-17T19:30:08.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:08.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:08.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:30:08.246 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:30:08.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:08.246 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:08.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:09.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:09.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:09.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:30:09.588 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:30:09.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:09.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:09.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:10.556 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:10.557+0000 7f9835d757c0 -1 osd.0 111 log_to_monitors true 2024-09-17T19:30:10.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:10.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:10.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:30:10.907 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:30:10.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:10.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:11.322 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 116 up_thru 107 down_at 112 last_clean_interval [100,111) [v2:127.0.0.1:6802/1702667797,v1:127.0.0.1:6803/1702667797] [v2:127.0.0.1:6804/1702667797,v1:127.0.0.1:6805/1702667797] exists,up e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:30:11.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:30:11.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:30:11.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:30:11.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:30:11.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:30:11.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:30:11.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:30:11.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:30:11.323 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:30:11.323 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:30:11.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:30:11.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:30:11.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:30:11.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:30:11.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:30:11.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:30:11.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:30:11.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:30:11.438 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:30:11.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:30:11.902 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:30:11.902 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:30:11.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:30:11.902 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:30:11.902 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:30:12.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=498216206338 2024-09-17T19:30:12.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 498216206338 2024-09-17T19:30:12.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338' 2024-09-17T19:30:12.141 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:30:12.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:30:12.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500678 2024-09-17T19:30:12.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500678 2024-09-17T19:30:12.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500678' 2024-09-17T19:30:12.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:30:12.290 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:30:12.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318290 2024-09-17T19:30:12.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318290 2024-09-17T19:30:12.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-498216206338 1-459561500678 2-352187318290' 2024-09-17T19:30:12.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:12.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-498216206338 2024-09-17T19:30:12.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:12.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:30:12.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-498216206338 2024-09-17T19:30:12.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:12.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=498216206338 2024-09-17T19:30:12.455 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-09-17T19:30:12.455 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 498216206338 2024-09-17T19:30:12.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:30:12.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 498216206338 2024-09-17T19:30:12.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:30:13.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:30:13.777 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:30:14.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 498216206338 -lt 498216206338 2024-09-17T19:30:14.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:14.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500678 2024-09-17T19:30:14.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:14.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:30:14.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500678 2024-09-17T19:30:14.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:14.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500678 2024-09-17T19:30:14.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500678' 2024-09-17T19:30:14.104 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 459561500678 2024-09-17T19:30:14.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:30:14.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500678 -lt 459561500678 2024-09-17T19:30:14.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:14.415 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318290 2024-09-17T19:30:14.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:14.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:30:14.418 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318290 2024-09-17T19:30:14.418 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:14.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318290 2024-09-17T19:30:14.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318290' 2024-09-17T19:30:14.421 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 352187318290 2024-09-17T19:30:14.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:30:14.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318290 -lt 352187318290 2024-09-17T19:30:14.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:30:14.764 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:14.764 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:30:15.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:30:15.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:30:15.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:30:15.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:30:15.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:15.522 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:30:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:30:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:30:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:30:15.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:30:15.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:30:15.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:30:15.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:30:15.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:30:15.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:30:15.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:30:16.238 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:30:19.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:30:19.133 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:30:19.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:30:19.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:30:19.137 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:30:19.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:30:19.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:30:19.139 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:30:19.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:30:19.171 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:19.169+0000 7fdb5deba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:19.174 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:19.173+0000 7fdb5deba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:19.175 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:19.173+0000 7fdb5deba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:19.475 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:30:19.476 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:30:19.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:19.476 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:19.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:20.455 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:20.453+0000 7fdb5deba7c0 -1 Falling back to public interface 2024-09-17T19:30:20.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:20.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:20.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:30:20.820 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:30:20.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:20.821 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:21.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:22.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:22.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:22.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:30:22.150 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:30:22.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:22.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:22.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:23.412 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:23.409+0000 7fdb5deba7c0 -1 osd.0 118 log_to_monitors true 2024-09-17T19:30:23.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:23.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:23.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:30:23.471 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:30:23.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:23.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:23.876 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 123 up_thru 116 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/517964597,v1:127.0.0.1:6803/517964597] [v2:127.0.0.1:6804/517964597,v1:127.0.0.1:6805/517964597] exists,up e509dc6b-96d3-48d2-a145-04014a106f18 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:30:23.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:30:23.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:30:23.878 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:30:23.878 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:30:23.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:30:23.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:30:23.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:30:23.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:30:23.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:30:23.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:30:23.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:30:23.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:30:23.995 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:30:24.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:30:24.339 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:30:24.339 INFO:tasks.workunit.client.0.smithi105.stderr:2' 2024-09-17T19:30:24.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:30:24.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:30:24.340 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:30:24.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-17T19:30:24.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-17T19:30:24.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410' 2024-09-17T19:30:24.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:30:24.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:30:24.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=459561500681 2024-09-17T19:30:24.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 459561500681 2024-09-17T19:30:24.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500681' 2024-09-17T19:30:24.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:30:24.805 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-17T19:30:25.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=352187318293 2024-09-17T19:30:25.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 352187318293 2024-09-17T19:30:25.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-528280977410 1-459561500681 2-352187318293' 2024-09-17T19:30:25.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:25.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-528280977410 2024-09-17T19:30:25.008 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:25.009 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:30:25.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-528280977410 2024-09-17T19:30:25.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:25.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-17T19:30:25.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-09-17T19:30:25.011 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 528280977410 2024-09-17T19:30:25.012 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:30:25.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 528280977410 2024-09-17T19:30:25.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:30:26.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:30:26.473 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:30:26.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-17T19:30:26.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:26.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-459561500681 2024-09-17T19:30:26.807 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:26.809 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:30:26.810 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-459561500681 2024-09-17T19:30:26.810 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:26.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=459561500681 2024-09-17T19:30:26.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 459561500681' 2024-09-17T19:30:26.812 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 459561500681 2024-09-17T19:30:26.812 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:30:27.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 459561500681 -lt 459561500681 2024-09-17T19:30:27.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:30:27.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-352187318293 2024-09-17T19:30:27.148 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:30:27.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-17T19:30:27.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-352187318293 2024-09-17T19:30:27.151 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:30:27.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=352187318293 2024-09-17T19:30:27.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 352187318293' 2024-09-17T19:30:27.152 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.2 seq 352187318293 2024-09-17T19:30:27.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-17T19:30:27.473 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 352187318294 -lt 352187318293 2024-09-17T19:30:27.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:30:27.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:27.474 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:27.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:30:27.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:30:27.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:30:27.876 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:30:27.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:30:27.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:30:27.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:30:27.877 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:30:28.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:30:28.235 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:30:28.236 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:28.236 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:28.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:30:28.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' -1 2024-09-17T19:30:28.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:30:28.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=5 2024-09-17T19:30:28.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:30:28.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:30:28.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:30:28.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:30:28.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:30:28.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:30:28.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:30:28.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:30:28.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:30:28.718 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:30:29.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=5 2024-09-17T19:30:29.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:30:29.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:29.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:29.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 = 6 2024-09-17T19:30:29.495 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 5 '!=' 5 2024-09-17T19:30:29.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:30:29.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:30:29.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:30:29.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:30:29.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:30:29.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:30:29.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:30:29.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:30:29.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:30:29.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:30:29.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:30:29.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:30:30.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:30:30.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:30:30.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:30:30.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:30:30.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:30:30.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:30:30.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:30:30.122 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:30:30.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:30:30.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:30:30.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:30:30.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:30:30.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:30:30.849 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:30:30.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:30:30.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-09-17T19:30:30.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-09-17T19:30:30.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2024-09-17T19:30:30.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2024-09-17T19:30:30.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2024-09-17T19:30:30.851 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:30:31.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-09-17T19:30:31.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-09-17T19:30:31.202 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-17T19:30:31.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-17T19:30:31.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:31.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:31.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:31.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:31.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:31.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-17T19:30:31.805 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-09-17T19:30:31.822 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:31.823 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:31.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:31.824 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:32.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:32.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:33.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:33.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:33.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:30:33.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:33.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:33.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:33.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:33.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:33.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:34.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:34.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:34.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:30:34.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:34.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:34.462 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:34.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:34.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:34.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:35.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:35.778 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:35.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:30:35.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:35.779 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:35.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:35.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:36.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:36.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:37.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:37.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:37.094 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:30:37.094 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:37.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:37.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:37.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:37.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:37.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:38.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:38.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:38.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:30:38.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:38.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:30:38.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:38.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:30:38.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:30:32.402482+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:38.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:30:38.746 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-17T19:30:38.780 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-17T19:30:38.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-09-17T19:30:38.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-17T19:30:38.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-17T19:30:38.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-17T19:30:38.864 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-17T19:30:38.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-09-17T19:30:38.891 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:30:38.891 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:30:38.891 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:30:38.891 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:30:38.891 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:30:38.892 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-17T19:30:38.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:30:38.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:30:38.971 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:30:38.972 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:30:38.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-09-17T19:30:38.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-09-17T19:30:38.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-09-17T19:30:38.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-17T19:30:38.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:30:38.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:38.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:30:38.974 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:38.975 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:30:39.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:39.296 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-17T19:30:39.586 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-09-17T19:30:39.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-17T19:27:31.381675+0000 last_deep_scrub_stamp 2024-09-17T19:30:39.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:30:39.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:39.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-17T19:30:39.601 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:30:39.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:39.602 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:30:39.602 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:39.602 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:30:39.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:39.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:30:39.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:39.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:40.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:40.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:40.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:30:40.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:40.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:30:40.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:40.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:30:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:27:31.381675+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:41.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:30:42.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:30:42.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:30:42.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:30:42.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:30:42.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:30:42.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:30:42.256 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:30:42.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:30:40.490007+0000 '>' 2024-09-17T19:27:31.381675+0000 2024-09-17T19:30:42.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:30:42.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-09-17T19:30:42.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-09-17T19:30:42.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-09-17T19:30:42.639 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-17T19:30:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-09-17T19:30:42.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-09-17T19:30:42.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:30:42.722 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:30:42.723 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:30:42.723 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:30:42.723 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:30:42.723 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T19:30:42.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:30:42.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:30:42.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-09-17T19:30:42.763 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:30:42.763 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-17T19:30:42.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-09-17T19:30:42.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:30:42.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:30:42.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:30:42.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-09-17T19:30:42.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-09-17T19:30:42.806 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-09-17T19:30:43.243 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'ecpool' removed 2024-09-17T19:30:43.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-09-17T19:30:43.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:30:43.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:30:43.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:30:43.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:30:43.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:30:43.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:30:43.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:30:43.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:30:43.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:30:43.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:30:43.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:30:43.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:30:43.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:30:43.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:30:43.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:30:43.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:30:43.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:30:43.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:30:43.382 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:30:43.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:30:43.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:30:43.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:30:43.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:30:43.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:43.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:43.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:30:43.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:30:43.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:30:43.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:30:43.417 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:30:43.417 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:30:43.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:30:43.418 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:30:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:30:43.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:30:43.419 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:30:43.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:30:43.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:30:43.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:30:43.421 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:30:43.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:30:43.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-17T19:30:43.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:30:43.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:30:43.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:43.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:43.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:30:43.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:30:43.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:30:43.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2024-09-17T19:30:43.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-17T19:30:43.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:43.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:43.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.46548 2024-09-17T19:30:43.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-17T19:30:43.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2024-09-17T19:30:43.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-09-17T19:30:43.895 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-17T19:30:43.895 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:30:43.896 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:30:43.896 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:30:43.896 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:43.896 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:43.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:43.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-17T19:30:44.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-17T19:30:44.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-17T19:30:44.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:30:44.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:30:44.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-17T19:30:44.117 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-17T19:30:44.117 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:30:44.117 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:30:44.117 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:30:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:30:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:30:44.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:30:44.119 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get fsid 2024-09-17T19:30:44.233 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-17T19:30:44.233 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-17T19:30:44.233 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-17T19:30:44.233 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-17T19:30:44.234 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-17T19:30:44.234 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-17T19:30:44.234 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-17T19:30:44.234 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-17T19:30:44.234 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:30:44.235 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:44.235 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:44.235 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-mon.a.asok 2024-09-17T19:30:44.235 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-17T19:30:44.235 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.46548/ceph-mon.a.asok config get mon_host 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2024-09-17T19:30:44.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-17T19:30:45.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-17T19:30:45.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:30:45.187 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:30:45.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:30:45.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:45.188 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:45.188 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:45.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:30:45.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:30:45.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:30:45.223 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:30:45.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:30:45.225 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:30:45.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:30:45.225 INFO:tasks.workunit.client.0.smithi105.stdout:add osd0 c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:30:45.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c8f8e2c8-d609-4274-a730-187075fe3e68' 2024-09-17T19:30:45.226 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:30:45.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDl2OlmclRxDhAAD80QyCfzkURFjfYvFCYuww== 2024-09-17T19:30:45.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDl2OlmclRxDhAAD80QyCfzkURFjfYvFCYuww=="}' 2024-09-17T19:30:45.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c8f8e2c8-d609-4274-a730-187075fe3e68 -i td/osd-scrub-repair/0/new.json 2024-09-17T19:30:45.501 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:30:45.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2024-09-17T19:30:45.521 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDl2OlmclRxDhAAD80QyCfzkURFjfYvFCYuww== --osd-uuid c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:30:45.552 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:45.548+0000 7ff0823967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:45.554 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:45.552+0000 7ff0823967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:45.555 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:45.552+0000 7ff0823967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:45.556 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:45.552+0000 7ff0823967c0 -1 bdev(0x56156c045c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2024-09-17T19:30:45.556 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:45.556+0000 7ff0823967c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-09-17T19:30:51.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-09-17T19:30:51.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:30:51.899 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd0 key to auth repository 2024-09-17T19:30:51.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-17T19:30:51.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:30:52.347 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.0 2024-09-17T19:30:52.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-17T19:30:52.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:30:52.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:30:52.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:30:52.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:30:52.380 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:52.376+0000 7f246f7317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:52.407 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:52.404+0000 7f246f7317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:52.409 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:52.408+0000 7f246f7317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:30:52.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-17T19:30:52.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:30:52.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:30:52.682 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:30:52.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:30:52.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:30:52.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:52.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:30:52.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:52.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:53.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:53.439 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:53.436+0000 7f246f7317c0 -1 Falling back to public interface 2024-09-17T19:30:54.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:54.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:54.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:30:54.022 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:30:54.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:54.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:54.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:55.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:55.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:55.340 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:30:55.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:30:55.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:55.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:55.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:56.155 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:30:56.152+0000 7f246f7317c0 -1 osd.0 0 log_to_monitors true 2024-09-17T19:30:56.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:56.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:56.667 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:30:56.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:30:56.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:56.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:57.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:58.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:58.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:58.117 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:30:58.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:30:58.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:58.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:58.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:30:59.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:30:59.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:30:59.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:30:59.450 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:30:59.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:30:59.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:30:59.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:00.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:00.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:00.774 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:31:00.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:31:00.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:00.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:01.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:02.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:02.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:02.167 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:31:02.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:31:02.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:02.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:02.490 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:03.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:03.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:03.493 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:31:03.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:31:03.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:03.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:03.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:04.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:04.985 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:04.985 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:31:04.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:31:04.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:04.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:05.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:06.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:06.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:06.323 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:31:06.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:31:06.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:06.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:06.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:07.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:07.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:07.754 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:31:07.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:31:07.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:07.755 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:08.109 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:09.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:09.112 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:31:09.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:09.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:31:09.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:09.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:09.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:10.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:10.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:10.543 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:31:10.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:31:10.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:10.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:11.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:12.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:12.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:12.136 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:31:12.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:31:12.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:12.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:12.944 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:13.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:13.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:13.946 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:31:13.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:31:13.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:13.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:14.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:15.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:15.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:15.750 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:31:15.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:31:15.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:15.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:16.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:17.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:17.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:17.414 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:31:17.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:31:17.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:17.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:18.116 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:19.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:19.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:19.118 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:31:19.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:31:19.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:19.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:19.657 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:20.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:20.660 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:20.660 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:31:20.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:31:20.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:20.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:31:20.992 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3389334676,v1:127.0.0.1:6803/3389334676] [v2:127.0.0.1:6804/3389334676,v1:127.0.0.1:6805/3389334676] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-17T19:31:20.995 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:31:20.996 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-17T19:31:20.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:31:20.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:31:20.997 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:31:20.997 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:31:20.997 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:31:20.997 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:31:20.999 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:31:21.000 INFO:tasks.workunit.client.0.smithi105.stdout:add osd1 ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:31:21.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ece21a65-749d-4e73-a3c1-8b3d29b5ecb6' 2024-09-17T19:31:21.001 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-17T19:31:21.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAJ2elmkOY8ARAANBfMjJMbieI8adxShnTOkw== 2024-09-17T19:31:21.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAJ2elmkOY8ARAANBfMjJMbieI8adxShnTOkw=="}' 2024-09-17T19:31:21.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 -i td/osd-scrub-repair/1/new.json 2024-09-17T19:31:21.386 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:31:21.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2024-09-17T19:31:21.405 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJ2elmkOY8ARAANBfMjJMbieI8adxShnTOkw== --osd-uuid ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:31:21.438 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:21.436+0000 7f1ea28447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:31:21.471 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:21.468+0000 7f1ea28447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:31:21.472 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:21.472+0000 7f1ea28447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:31:21.472 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:21.472+0000 7f1ea28447c0 -1 bdev(0x55b2edcec800 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2024-09-17T19:31:21.473 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:21.472+0000 7f1ea28447c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-09-17T19:31:27.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-09-17T19:31:27.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-17T19:31:27.864 INFO:tasks.workunit.client.0.smithi105.stdout:adding osd1 key to auth repository 2024-09-17T19:31:27.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-17T19:31:27.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-17T19:31:28.280 INFO:tasks.workunit.client.0.smithi105.stdout:start osd.1 2024-09-17T19:31:28.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-17T19:31:28.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:31:28.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-17T19:31:28.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-17T19:31:28.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-17T19:31:28.313 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:28.311+0000 7f38dfa5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:31:28.315 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:28.311+0000 7f38dfa5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:31:28.316 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:28.315+0000 7f38dfa5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:28.609 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:31:28.610 INFO:tasks.workunit.client.0.smithi105.stdout:0 2024-09-17T19:31:28.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:28.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:28.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:29.755 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:29.751+0000 7f38dfa5b7c0 -1 Falling back to public interface 2024-09-17T19:31:29.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:29.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:29.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:31:29.950 INFO:tasks.workunit.client.0.smithi105.stdout:1 2024-09-17T19:31:29.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:29.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:30.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:31.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:31.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:31.285 INFO:tasks.workunit.client.0.smithi105.stdout:2 2024-09-17T19:31:31.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:31:31.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:31.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:31.618 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:32.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:32.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:32.621 INFO:tasks.workunit.client.0.smithi105.stdout:3 2024-09-17T19:31:32.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:31:32.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:32.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:32.877 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:31:32.875+0000 7f38dfa5b7c0 -1 osd.1 0 log_to_monitors true 2024-09-17T19:31:33.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:34.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:34.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:34.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:31:34.011 INFO:tasks.workunit.client.0.smithi105.stdout:4 2024-09-17T19:31:34.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:34.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:34.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:35.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:35.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:35.362 INFO:tasks.workunit.client.0.smithi105.stdout:5 2024-09-17T19:31:35.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-17T19:31:35.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:35.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:35.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:36.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:36.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:36.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-17T19:31:36.789 INFO:tasks.workunit.client.0.smithi105.stdout:6 2024-09-17T19:31:36.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:36.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:37.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:38.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:38.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:38.189 INFO:tasks.workunit.client.0.smithi105.stdout:7 2024-09-17T19:31:38.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-17T19:31:38.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:38.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:38.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:39.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:39.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:39.622 INFO:tasks.workunit.client.0.smithi105.stdout:8 2024-09-17T19:31:39.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-17T19:31:39.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:39.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:40.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:41.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:41.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:41.043 INFO:tasks.workunit.client.0.smithi105.stdout:9 2024-09-17T19:31:41.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-17T19:31:41.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:41.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:41.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:42.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:42.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:42.367 INFO:tasks.workunit.client.0.smithi105.stdout:10 2024-09-17T19:31:42.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-17T19:31:42.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:42.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:42.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:43.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:43.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:43.889 INFO:tasks.workunit.client.0.smithi105.stdout:11 2024-09-17T19:31:43.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-17T19:31:43.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:43.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:44.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:45.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:45.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:45.227 INFO:tasks.workunit.client.0.smithi105.stdout:12 2024-09-17T19:31:45.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-17T19:31:45.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:45.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:45.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:46.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:46.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:46.561 INFO:tasks.workunit.client.0.smithi105.stdout:13 2024-09-17T19:31:46.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-17T19:31:46.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:46.562 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:47.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:48.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:48.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:48.152 INFO:tasks.workunit.client.0.smithi105.stdout:14 2024-09-17T19:31:48.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-17T19:31:48.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:48.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:48.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:49.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:49.926 INFO:tasks.workunit.client.0.smithi105.stdout:15 2024-09-17T19:31:49.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-17T19:31:49.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:49.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:50.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:51.691 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:51.692 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:51.692 INFO:tasks.workunit.client.0.smithi105.stdout:16 2024-09-17T19:31:51.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-17T19:31:51.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:51.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:52.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:53.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:53.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:53.508 INFO:tasks.workunit.client.0.smithi105.stdout:17 2024-09-17T19:31:53.508 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-17T19:31:53.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:53.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:54.071 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:55.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:55.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:55.074 INFO:tasks.workunit.client.0.smithi105.stdout:18 2024-09-17T19:31:55.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-17T19:31:55.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:55.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:55.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:56.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:56.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:56.519 INFO:tasks.workunit.client.0.smithi105.stdout:19 2024-09-17T19:31:56.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 19 2024-09-17T19:31:56.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:56.519 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:56.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:31:57.932 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:31:57.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:31:57.933 INFO:tasks.workunit.client.0.smithi105.stdout:20 2024-09-17T19:31:57.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 20 2024-09-17T19:31:57.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:31:57.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:31:58.263 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3023835040,v1:127.0.0.1:6811/3023835040] [v2:127.0.0.1:6812/3023835040,v1:127.0.0.1:6813/3023835040] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:31:58.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:31:58.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:31:58.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:31:58.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-09-17T19:31:58.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-17T19:31:58.583 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' does not exist 2024-09-17T19:31:58.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2024-09-17T19:31:58.599 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2024-09-17T19:31:59.038 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'rbd' created 2024-09-17T19:31:59.058 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:32:00.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2024-09-17T19:32:02.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-17T19:32:02.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:32:02.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:32:02.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:32:02.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:32:02.195 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:32:02.195 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:32:02.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:32:02.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:32:02.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:32:02.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:32:02.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:32:02.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:32:02.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:32:02.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:32:02.328 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:32:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:32:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:32:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:32:02.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:32:02.657 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:32:02.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836493 2024-09-17T19:32:02.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836493 2024-09-17T19:32:02.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493' 2024-09-17T19:32:02.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:32:02.817 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:32:02.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-17T19:32:02.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-17T19:32:02.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-09-17T19:32:02.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:32:02.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836493 2024-09-17T19:32:02.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:32:02.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:32:02.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836493 2024-09-17T19:32:02.963 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:32:02.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836493 2024-09-17T19:32:02.965 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836493 2024-09-17T19:32:02.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-17T19:32:02.965 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:32:03.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836493 2024-09-17T19:32:03.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:32:04.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:32:04.285 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:32:04.623 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836493 -lt 21474836493 2024-09-17T19:32:04.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:32:04.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-17T19:32:04.625 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:32:04.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:32:04.628 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:32:04.628 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-17T19:32:04.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-17T19:32:04.630 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672965 2024-09-17T19:32:04.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-17T19:32:04.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:32:04.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-17T19:32:04.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:32:04.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:32:04.967 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:32:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 4 == 0 2024-09-17T19:32:05.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:32:05.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:32:05.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:32:05.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:32:05.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:32:05.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:32:05.454 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:32:05.813 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:32:05.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:32:05.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:32:05.813 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:32:06.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 4 2024-09-17T19:32:06.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:32:06.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:32:06.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-09-17T19:32:06.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2024-09-17T19:32:06.629 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'foo' created 2024-09-17T19:32:06.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:32:07.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-09-17T19:32:07.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2024-09-17T19:32:08.292 INFO:tasks.workunit.client.0.smithi105.stderr:pool 'csr_pool' created 2024-09-17T19:32:08.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-17T19:32:09.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-17T19:32:09.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:32:09.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:32:09.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:32:09.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:32:09.322 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:32:09.322 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:32:09.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:32:09.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:32:09.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:32:09.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:32:09.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:32:09.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:32:09.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:32:09.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:32:09.458 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:32:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:32:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:32:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:32:09.790 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:32:09.790 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:32:09.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836495 2024-09-17T19:32:09.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836495 2024-09-17T19:32:09.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495' 2024-09-17T19:32:09.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:32:09.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:32:10.090 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672967 2024-09-17T19:32:10.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672967 2024-09-17T19:32:10.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836495 1-42949672967' 2024-09-17T19:32:10.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:32:10.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836495 2024-09-17T19:32:10.091 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:32:10.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:32:10.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836495 2024-09-17T19:32:10.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:32:10.095 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 21474836495 2024-09-17T19:32:10.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836495 2024-09-17T19:32:10.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-17T19:32:10.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:32:10.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-17T19:32:10.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:32:11.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:32:11.415 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:32:11.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836494 -lt 21474836495 2024-09-17T19:32:11.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:32:12.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:32:12.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:32:13.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836496 -lt 21474836495 2024-09-17T19:32:13.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:32:13.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672967 2024-09-17T19:32:13.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:32:13.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:32:13.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672967 2024-09-17T19:32:13.061 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:32:13.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672967 2024-09-17T19:32:13.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-17T19:32:13.062 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 42949672967 2024-09-17T19:32:13.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:32:13.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672967 -lt 42949672967 2024-09-17T19:32:13.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:32:13.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:32:13.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:32:13.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:32:13.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:32:13.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:32:13.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:32:13.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:32:13.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:32:13.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:32:13.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:32:14.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:32:14.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:32:14.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:32:14.069 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:32:14.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:32:14.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:32:14.462 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:32:14.463 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-17T19:32:14.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:14.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:14.465 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:14.879 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:14.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:15.423 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:15.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:15.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:15.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:15.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-09-17T19:32:15.897 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-09-17T19:32:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:16.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:16.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:16.468 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:16.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:17.012 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:17.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:17.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:17.034 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:17.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-09-17T19:32:17.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-09-17T19:32:17.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:17.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-17T19:32:17.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-09-17T19:32:17.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:17.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:17.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-09-17T19:32:17.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:17.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:17.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:17.972 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:17.990 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:18.567 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:18.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-09-17T19:32:19.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:19.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:19.553 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:19.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:20.037 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:20.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:20.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:20.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:20.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-09-17T19:32:20.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-09-17T19:32:20.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:20.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-17T19:32:20.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-09-17T19:32:20.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:20.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:20.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-09-17T19:32:20.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:20.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:20.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:21.074 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:21.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:21.660 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:21.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:21.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:21.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:21.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-09-17T19:32:22.140 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-09-17T19:32:22.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:22.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-17T19:32:22.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-09-17T19:32:22.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:22.250 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:22.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-09-17T19:32:22.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:22.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:22.251 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:22.671 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:22.690 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:23.181 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:23.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:23.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:23.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:23.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-09-17T19:32:23.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-09-17T19:32:23.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:23.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:23.697 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:24.251 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:24.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:24.795 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:24.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:24.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:24.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:25.008 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-09-17T19:32:25.186 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-09-17T19:32:25.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:25.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:25.294 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:25.748 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:25.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:26.367 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:26.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:26.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:26.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-09-17T19:32:26.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-09-17T19:32:27.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:27.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:27.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:27.521 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:27.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:28.039 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:28.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:28.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:28.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:28.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-09-17T19:32:28.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-09-17T19:32:28.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:28.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-17T19:32:28.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-09-17T19:32:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-09-17T19:32:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:28.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:29.051 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:29.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:29.586 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:29.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:29.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:29.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:29.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-09-17T19:32:30.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-09-17T19:32:30.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:30.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-17T19:32:30.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-09-17T19:32:30.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:30.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:30.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-09-17T19:32:30.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:30.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:30.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:30.623 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:30.643 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:31.183 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:31.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:31.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:31.207 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:31.396 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-09-17T19:32:31.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:31.732 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:32.245 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:32.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:32.763 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:32.785 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:32.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:32.786 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:33.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-09-17T19:32:33.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:33.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:33.741 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:33.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:34.234 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:34.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:34.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:34.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:34.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-09-17T19:32:34.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:34.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:34.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:35.237 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:35.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:35.756 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:35.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:35.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:35.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:36.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-09-17T19:32:36.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:36.793 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:36.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:37.328 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:37.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:37.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:37.348 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:37.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-09-17T19:32:37.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-09-17T19:32:37.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:37.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-17T19:32:37.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-09-17T19:32:37.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:37.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:37.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-09-17T19:32:37.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:37.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:37.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:38.289 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:38.309 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:38.841 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:38.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:38.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:38.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:39.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-09-17T19:32:39.230 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-09-17T19:32:39.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:39.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:39.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:39.768 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:39.787 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:40.362 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:40.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:40.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:40.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:40.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-09-17T19:32:40.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-09-17T19:32:40.894 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:40.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:41.341 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:41.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:41.850 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:41.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:41.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:41.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:42.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-09-17T19:32:42.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-09-17T19:32:42.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:42.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-17T19:32:42.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-09-17T19:32:42.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-09-17T19:32:42.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-09-17T19:32:42.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-09-17T19:32:42.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-09-17T19:32:42.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-09-17T19:32:42.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-09-17T19:32:42.929 INFO:tasks.workunit.client.0.smithi105.stderr:noscrub is set 2024-09-17T19:32:42.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-09-17T19:32:43.497 INFO:tasks.workunit.client.0.smithi105.stderr:nodeep-scrub is set 2024-09-17T19:32:43.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-09-17T19:32:43.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-09-17T19:32:43.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-09-17T19:32:43.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-09-17T19:32:43.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-09-17T19:32:44.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-09-17T19:32:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:1025+0 records in 2024-09-17T19:32:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:1025+0 records out 2024-09-17T19:32:44.118 INFO:tasks.workunit.client.0.smithi105.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00378209 s, 278 MB/s 2024-09-17T19:32:44.119 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-09-17T19:32:44.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-09-17T19:32:44.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-17T19:32:44.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-17T19:32:44.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-17T19:32:44.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:32:44.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-17T19:32:44.698 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-17T19:32:44.698 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-09-17T19:32:44.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2024-09-17T19:32:44.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2024-09-17T19:32:44.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-09-17T19:32:44.699 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-17T19:32:45.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-09-17T19:32:45.072 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-09-17T19:32:45.072 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:32:45.073 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:32:45.073 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:32:45.073 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:32:45.074 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:32:45.074 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:32:45.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-17T19:32:45.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.46548/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-17T19:32:45.182 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T19:32:45.182 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-17T19:32:45.183 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T19:32:45.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-09-17T19:32:45.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:32:45.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:32:45.197 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:32:45.197 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:32:45.197 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:32:45.198 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:32:45.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-09-17T19:32:45.198 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-09-17T19:32:45.308 INFO:tasks.workunit.client.0.smithi105.stdout:{ 2024-09-17T19:32:45.308 INFO:tasks.workunit.client.0.smithi105.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-09-17T19:32:45.308 INFO:tasks.workunit.client.0.smithi105.stdout:} 2024-09-17T19:32:45.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-17T19:32:45.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-17T19:32:45.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:32:45.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:32:45.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:32:45.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:32:45.324 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:32:45.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-17T19:32:08.049981+0000 2024-09-17T19:32:45.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-17T19:32:45.930 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-17T19:32:45.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-17T19:32:08.049981+0000 last_deep_scrub_stamp 2024-09-17T19:32:45.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:32:45.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:32:08.049981+0000 2024-09-17T19:32:45.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-17T19:32:45.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:32:45.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:32:45.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:32:45.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:32:45.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:32:45.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:32:45.949 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:32:46.270 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:08.049981+0000 '>' 2024-09-17T19:32:08.049981+0000 2024-09-17T19:32:46.271 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:32:47.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:32:47.272 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:32:47.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:32:47.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:32:47.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:32:47.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:32:47.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:32:47.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:08.049981+0000 '>' 2024-09-17T19:32:08.049981+0000 2024-09-17T19:32:47.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:32:48.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:32:48.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:32:48.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:32:48.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:32:48.696 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:32:48.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:32:48.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:32:49.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:47.366014+0000 '>' 2024-09-17T19:32:08.049981+0000 2024-09-17T19:32:49.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:32:49.025 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-09-17T19:32:49.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:32:49.026 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-09-17T19:32:49.027 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-09-17T19:32:49.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:32:49.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:32:49.030 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:32:49.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:32:49.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:32:49.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:32:49.031 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:32:49.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:32:49.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:32:50.995 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-09-17T19:32:51.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:32:51.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:32:51.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:32:51.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:32:51.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:32:51.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:32:51.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:32:51.583 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:32:51.583 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:32:51.583 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:32:51.583 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:32:51.583 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:32:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:32:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:32:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:32:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:32:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:32:51.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:32:51.586 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:32:51.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:32:51.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:32:51.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:32:51.618 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:32:51.618+0000 7f4e4c2087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:32:51.621 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:32:51.622+0000 7f4e4c2087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:32:51.623 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:32:51.622+0000 7f4e4c2087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:32:51.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:32:51.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:32:52.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:32:52.915 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:32:52.914+0000 7f4e4c2087c0 -1 Falling back to public interface 2024-09-17T19:32:53.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:32:53.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:32:53.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:32:53.284 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:32:53.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:32:53.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:32:53.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:32:54.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:32:54.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:32:54.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:32:54.613 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:32:54.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:32:54.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:32:54.931 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:32:55.873 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:32:55.874+0000 7f4e4c2087c0 -1 osd.1 64 log_to_monitors true 2024-09-17T19:32:55.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:32:55.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:32:55.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:32:55.933 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:32:55.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:32:55.934 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:32:56.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:32:57.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:32:57.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:32:57.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:32:57.314 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:32:57.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:32:57.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:32:57.663 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/3987921265,v1:127.0.0.1:6811/3987921265] [v2:127.0.0.1:6812/3987921265,v1:127.0.0.1:6813/3987921265] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:32:57.664 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:32:57.665 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:32:57.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:32:57.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:32:57.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:32:57.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:32:57.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:32:57.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:32:57.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:32:57.781 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:32:57.781 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:32:58.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:32:58.097 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:32:58.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:32:58.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:32:58.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:32:58.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-17T19:32:58.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-17T19:32:58.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-17T19:32:58.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:32:58.258 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:32:58.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776130 2024-09-17T19:32:58.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776130 2024-09-17T19:32:58.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-292057776130' 2024-09-17T19:32:58.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:32:58.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-17T19:32:58.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:32:58.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:32:58.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-17T19:32:58.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:32:58.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-17T19:32:58.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-17T19:32:58.428 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 21474836506 2024-09-17T19:32:58.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:32:58.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836506 2024-09-17T19:32:58.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:32:59.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:32:59.751 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:00.081 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-17T19:33:00.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:00.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776130 2024-09-17T19:33:00.082 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:00.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:33:00.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776130 2024-09-17T19:33:00.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:00.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776130 2024-09-17T19:33:00.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-09-17T19:33:00.087 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 292057776130 2024-09-17T19:33:00.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:33:00.389 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776130 -lt 292057776130 2024-09-17T19:33:00.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:33:00.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:00.390 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:00.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:00.783 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:01.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:33:01.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:01.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:01.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:01.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:33:01.505 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:33:01.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:33:01.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:33:01.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-09-17T19:33:01.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:33:01.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:33:01.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:33:01.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:33:01.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:33:01.511 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:33:01.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:33:01.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:01.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:33:01.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:33:01.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:33:01.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:33:01.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:33:01.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:04.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:33:04.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:33:04.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:33:04.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:33:04.171 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:33:04.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:04.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:33:04.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:33:04.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:33:04.176 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:33:04.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:33:04.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:33:04.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:33:04.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:33:04.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:33:04.178 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:33:04.209 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:04.209+0000 7f2b4213a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:04.452 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:04.213+0000 7f2b4213a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:04.452 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:04.213+0000 7f2b4213a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:04.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:33:04.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:04.544 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:04.878 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:05.487 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:05.485+0000 7f2b4213a7c0 -1 Falling back to public interface 2024-09-17T19:33:05.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:05.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:05.880 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:33:05.880 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:33:05.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:05.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:06.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:07.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:07.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:07.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:33:07.204 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:33:07.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:07.204 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:07.527 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:08.519 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:08.517+0000 7f2b4213a7c0 -1 osd.0 70 log_to_monitors true 2024-09-17T19:33:08.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:08.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:08.528 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:33:08.528 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:33:08.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:08.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 74 up_thru 69 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/3863150951,v1:127.0.0.1:6803/3863150951] [v2:127.0.0.1:6804/3863150951,v1:127.0.0.1:6805/3863150951] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:33:08.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:33:08.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:33:08.955 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:33:08.955 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:33:08.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:33:08.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:33:08.956 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:33:09.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:33:09.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:33:09.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:33:09.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:33:09.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:33:09.065 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:33:09.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:33:09.451 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:33:09.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:33:09.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:09.451 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:33:09.602 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579906 2024-09-17T19:33:09.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579906 2024-09-17T19:33:09.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579906' 2024-09-17T19:33:09.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:09.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:33:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=292057776133 2024-09-17T19:33:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 292057776133 2024-09-17T19:33:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579906 1-292057776133' 2024-09-17T19:33:09.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:09.766 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-317827579906 2024-09-17T19:33:09.767 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:09.767 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:33:09.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-317827579906 2024-09-17T19:33:09.769 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579906 2024-09-17T19:33:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-09-17T19:33:09.770 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 317827579906 2024-09-17T19:33:09.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:10.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579906 -lt 317827579906 2024-09-17T19:33:10.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:10.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-292057776133 2024-09-17T19:33:10.092 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:10.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:33:10.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-292057776133 2024-09-17T19:33:10.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:10.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=292057776133 2024-09-17T19:33:10.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 292057776133' 2024-09-17T19:33:10.097 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 292057776133 2024-09-17T19:33:10.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:33:10.504 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 292057776133 -lt 292057776133 2024-09-17T19:33:10.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:33:10.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:10.506 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:10.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:33:10.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:10.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:10.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:10.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:10.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:10.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:10.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:11.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:33:11.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:11.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:11.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:11.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:33:11.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:33:11.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:33:11.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:33:11.641 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:33:11.642 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:33:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:33:11.742 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:11.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:11.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:11.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:11.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:11.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:11.744 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:12.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:33:12.045 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:12.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:12.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:12.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:33:12.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:33:12.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:33:12.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:33:12.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:33:12.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:33:12.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:33:12.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:33:12.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:33:12.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:33:12.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:33:12.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:33:12.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:33:12.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:33:13.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:33:13.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:13.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:13.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:13.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:13.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:13.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:13.015 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:13.326 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:33:13.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:13.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:13.327 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:13.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:33:13.728 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:33:13.729 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:33:14.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:33:14.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:33:14.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-17T19:33:14.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:33:14.118 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T19:33:14.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:33:14.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:14.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:14.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:14.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:14.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:14.520 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:14.520 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:14.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:33:14.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:14.835 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:14.835 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:33:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:33:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:33:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:33:15.220 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-09-17T19:33:15.221 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-09-17T19:33:15.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:33:15.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:33:15.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-09-17T19:33:15.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:33:15.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:33:15.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:33:15.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:33:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:33:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:33:15.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:33:15.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:33:15.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-09-17T19:33:17.145 INFO:tasks.workunit.client.0.smithi105.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-09-17T19:33:17.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:33:17.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:33:17.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:33:17.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:33:17.682 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:33:17.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:17.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:17.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:33:17.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:33:17.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:33:17.685 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:33:17.686 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:33:17.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:33:17.686 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:33:17.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:33:17.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:33:17.723 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:17.721+0000 7ff1541f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:17.726 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:17.725+0000 7ff1541f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:17.728 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:17.725+0000 7ff1541f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:18.017 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:18.018 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:18.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:19.227 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:19.225+0000 7ff1541f27c0 -1 Falling back to public interface 2024-09-17T19:33:19.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:19.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:19.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:33:19.345 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:33:19.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:19.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:19.679 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:20.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:20.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:20.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:33:20.681 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:33:20.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:20.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:21.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:21.979 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:21.977+0000 7ff1541f27c0 -1 osd.1 75 log_to_monitors true 2024-09-17T19:33:22.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:22.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:22.015 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:33:22.015 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:33:22.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:22.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:22.497 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 79 up_thru 74 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/338234323,v1:127.0.0.1:6811/338234323] [v2:127.0.0.1:6812/338234323,v1:127.0.0.1:6813/338234323] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:33:22.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:33:22.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:33:22.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:33:22.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:33:22.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:33:22.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:33:22.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:33:22.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:33:22.499 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:33:22.499 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:33:22.500 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:33:22.500 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:33:22.500 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:33:22.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:33:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:33:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:33:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:33:22.611 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:33:22.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:33:22.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:33:22.991 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:33:22.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:33:22.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:22.992 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:33:23.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=317827579909 2024-09-17T19:33:23.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 317827579909 2024-09-17T19:33:23.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579909' 2024-09-17T19:33:23.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:23.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:33:23.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416386 2024-09-17T19:33:23.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416386 2024-09-17T19:33:23.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-09-17T19:33:23.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:23.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-317827579909 2024-09-17T19:33:23.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:23.265 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:33:23.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-317827579909 2024-09-17T19:33:23.266 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:23.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=317827579909 2024-09-17T19:33:23.268 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-09-17T19:33:23.268 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 317827579909 2024-09-17T19:33:23.268 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:23.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579908 -lt 317827579909 2024-09-17T19:33:23.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:33:24.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:33:24.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:24.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 317827579910 -lt 317827579909 2024-09-17T19:33:24.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:24.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-339302416386 2024-09-17T19:33:24.912 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:24.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:33:24.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-339302416386 2024-09-17T19:33:24.914 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:24.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416386 2024-09-17T19:33:24.915 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-09-17T19:33:24.915 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 339302416386 2024-09-17T19:33:24.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:33:25.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416386 -lt 339302416386 2024-09-17T19:33:25.259 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:33:25.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:25.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:25.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:33:25.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:25.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:25.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:25.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:25.659 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:25.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:25.660 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:25.979 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:33:25.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:25.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:25.980 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:26.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:33:26.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:33:26.342 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:33:26.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:33:26.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-09-17T19:33:26.343 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:26.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:33:26.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:33:26.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:33:26.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:26.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:33:26.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:33:26.655 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:33:26.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:33:26.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:33:26.656 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-09-17T19:33:28.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:33:28.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:33:28.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:33:28.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:33:28.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:33:28.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:33:28.836 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:28.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:28.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:33:28.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:33:28.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:33:28.839 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:33:28.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:33:28.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:33:28.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:33:28.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:33:28.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:33:28.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:33:28.872 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:28.873+0000 7f14c0dfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:28.875 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:28.873+0000 7f14c0dfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:28.876 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:28.877+0000 7f14c0dfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:29.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:33:29.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:29.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:29.514 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:29.951 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:29.949+0000 7f14c0dfe7c0 -1 Falling back to public interface 2024-09-17T19:33:30.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:30.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:30.516 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:33:30.516 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:33:30.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:30.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:30.835 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:33:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:33:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:31.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:32.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:32.873 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:32.873+0000 7f14c0dfe7c0 -1 osd.0 80 log_to_monitors true 2024-09-17T19:33:33.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:33.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:33.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:33:33.182 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:33:33.182 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:33.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:33.517 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 84 up_thru 79 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/108656927,v1:127.0.0.1:6803/108656927] [v2:127.0.0.1:6804/108656927,v1:127.0.0.1:6805/108656927] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:33:33.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:33:33.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:33:33.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:33:33.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:33:33.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:33:33.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:33:33.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:33:33.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:33:33.519 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:33:33.519 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:33:33.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:33:33.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:33:33.519 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:33:33.628 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:33:33.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:33:33.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:33:33.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:33:33.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:33:33.629 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:33:33.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:33:33.961 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:33:33.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:33:33.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:33.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:33:34.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252866 2024-09-17T19:33:34.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252866 2024-09-17T19:33:34.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252866' 2024-09-17T19:33:34.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:34.155 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:33:34.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=339302416389 2024-09-17T19:33:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 339302416389 2024-09-17T19:33:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-09-17T19:33:34.378 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:34.378 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-360777252866 2024-09-17T19:33:34.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:34.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:33:34.380 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-360777252866 2024-09-17T19:33:34.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:34.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252866 2024-09-17T19:33:34.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-09-17T19:33:34.382 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 360777252866 2024-09-17T19:33:34.382 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:34.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 360777252866 2024-09-17T19:33:34.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:33:35.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:33:35.710 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:36.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252866 -lt 360777252866 2024-09-17T19:33:36.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:36.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-339302416389 2024-09-17T19:33:36.038 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:36.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:33:36.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-339302416389 2024-09-17T19:33:36.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:36.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=339302416389 2024-09-17T19:33:36.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-09-17T19:33:36.043 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 339302416389 2024-09-17T19:33:36.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:33:36.347 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 339302416389 -lt 339302416389 2024-09-17T19:33:36.348 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:33:36.349 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:36.349 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:36.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:36.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:37.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:33:37.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:37.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:37.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:33:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:33:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:33:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:33:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:33:37.470 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:33:37.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:33:37.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:37.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:37.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:37.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:37.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:37.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:37.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:37.879 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:33:37.880 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:37.880 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:37.880 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:38.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:33:38.275 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T19:33:38.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:33:38.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:33:38.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:33:38.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:33:38.276 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:33:38.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:33:38.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:33:38.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:33:38.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:33:38.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:33:38.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:33:38.676 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:33:38.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:33:38.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:38.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:38.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:38.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:38.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:38.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:38.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:39.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:33:39.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:39.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:39.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:39.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:33:39.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:33:39.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:33:39.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:33:39.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-09-17T19:33:39.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-17T19:33:39.594 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:33:39.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:33:39.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:33:39.905 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-09-17T19:33:39.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:33:39.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:33:39.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:33:39.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:33:39.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:33:39.906 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-09-17T19:33:42.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:33:42.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:33:42.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:33:42.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:33:42.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:33:42.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:33:42.158 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:33:42.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:33:42.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:33:42.159 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:33:42.159 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:33:42.159 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:33:42.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:42.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:42.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:33:42.161 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:33:42.162 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:33:42.163 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:33:42.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:33:42.163 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:33:42.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:33:42.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:33:42.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:33:42.164 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:33:42.194 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:42.193+0000 7fc04bc3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:42.197 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:42.197+0000 7fc04bc3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:42.198 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:42.197+0000 7fc04bc3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:42.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:33:42.499 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:33:42.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:42.499 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:42.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:43.251 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:43.248+0000 7fc04bc3b7c0 -1 Falling back to public interface 2024-09-17T19:33:43.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:43.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:43.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:33:43.837 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:33:43.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:43.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:44.172 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:33:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:33:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:45.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:45.496 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:46.192 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:46.188+0000 7fc04bc3b7c0 -1 osd.1 86 log_to_monitors true 2024-09-17T19:33:46.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:46.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:46.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:33:46.498 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:33:46.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:46.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 90 up_thru 85 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/3092729101,v1:127.0.0.1:6811/3092729101] [v2:127.0.0.1:6812/3092729101,v1:127.0.0.1:6813/3092729101] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:33:46.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:33:46.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:33:46.929 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:33:46.930 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:33:46.930 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:33:46.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:33:46.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:33:46.930 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:33:47.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:33:47.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:33:47.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:33:47.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:33:47.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:33:47.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:33:47.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:33:47.420 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:33:47.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:33:47.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:47.420 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:33:47.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=360777252869 2024-09-17T19:33:47.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 360777252869 2024-09-17T19:33:47.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252869' 2024-09-17T19:33:47.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:47.675 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:33:47.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056642 2024-09-17T19:33:47.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056642 2024-09-17T19:33:47.823 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-09-17T19:33:47.824 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:47.825 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-360777252869 2024-09-17T19:33:47.825 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:47.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:33:47.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-360777252869 2024-09-17T19:33:47.827 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:47.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=360777252869 2024-09-17T19:33:47.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-09-17T19:33:47.828 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 360777252869 2024-09-17T19:33:47.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:48.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252869 -lt 360777252869 2024-09-17T19:33:48.160 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-386547056642 2024-09-17T19:33:48.161 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:48.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:33:48.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-386547056642 2024-09-17T19:33:48.165 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:48.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056642 2024-09-17T19:33:48.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-09-17T19:33:48.166 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 386547056642 2024-09-17T19:33:48.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:33:48.498 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056642 -lt 386547056642 2024-09-17T19:33:48.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:33:48.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:48.499 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:48.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:33:48.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:48.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:48.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:48.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:48.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:48.897 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:48.898 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:33:49.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:33:49.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:33:49.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:49.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:49.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:33:49.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:33:49.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:33:49.612 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:33:49.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-09-17T19:33:49.613 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:33:49.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:33:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:33:49.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:33:49.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:33:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:33:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:33:49.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:33:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:33:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-17T19:33:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:33:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:33:49.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:33:49.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:33:49.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:33:49.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-09-17T19:33:51.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:33:51.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:33:51.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:33:51.863 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:33:51.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:33:51.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:33:51.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:33:51.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:33:51.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:33:51.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:33:51.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:33:51.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:33:51.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:33:51.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:33:51.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:33:51.868 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:33:51.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:33:51.868 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:33:51.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:33:51.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:33:51.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:33:51.869 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:33:51.901 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:51.900+0000 7f3d2021e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:51.903 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:51.900+0000 7f3d2021e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:51.905 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:51.904+0000 7f3d2021e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:33:52.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:33:52.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:33:52.208 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:52.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:52.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:53.175 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:53.172+0000 7f3d2021e7c0 -1 Falling back to public interface 2024-09-17T19:33:53.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:53.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:53.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:33:53.536 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:33:53.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:53.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:53.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:54.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:54.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:54.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:33:54.850 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:33:54.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:54.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:55.185 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:33:55.886 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:33:55.884+0000 7f3d2021e7c0 -1 osd.0 91 log_to_monitors true 2024-09-17T19:33:56.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:33:56.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:33:56.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:33:56.187 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:33:56.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:33:56.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:33:56.587 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 95 up_thru 90 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/1139667945,v1:127.0.0.1:6803/1139667945] [v2:127.0.0.1:6804/1139667945,v1:127.0.0.1:6805/1139667945] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:33:56.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:33:56.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:33:56.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:33:56.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:33:56.587 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:33:56.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:33:56.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:33:56.588 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:33:56.589 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:33:56.589 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:33:56.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:33:56.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:33:56.589 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:33:56.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:33:56.707 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:33:56.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:33:56.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:33:56.708 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:33:56.708 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:33:57.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:33:57.035 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:33:57.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:33:57.035 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:57.035 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:33:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893122 2024-09-17T19:33:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893122 2024-09-17T19:33:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893122' 2024-09-17T19:33:57.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:33:57.193 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:33:57.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=386547056645 2024-09-17T19:33:57.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 386547056645 2024-09-17T19:33:57.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893122 1-386547056645' 2024-09-17T19:33:57.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:57.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-408021893122 2024-09-17T19:33:57.337 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:57.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:33:57.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-408021893122 2024-09-17T19:33:57.339 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893122 2024-09-17T19:33:57.341 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2024-09-17T19:33:57.341 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 408021893122 2024-09-17T19:33:57.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:57.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 360777252870 -lt 408021893122 2024-09-17T19:33:57.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:33:58.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:33:58.723 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:33:59.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893122 -lt 408021893122 2024-09-17T19:33:59.055 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:33:59.055 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-386547056645 2024-09-17T19:33:59.056 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:33:59.057 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:33:59.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-386547056645 2024-09-17T19:33:59.058 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:33:59.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=386547056645 2024-09-17T19:33:59.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-09-17T19:33:59.059 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 386547056645 2024-09-17T19:33:59.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:33:59.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 386547056645 -lt 386547056645 2024-09-17T19:33:59.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:33:59.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:33:59.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:33:59.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:33:59.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:33:59.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:33:59.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:33:59.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:33:59.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:33:59.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:33:59.804 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:00.157 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:34:00.158 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:00.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:00.159 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:00.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:34:00.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-17T19:34:00.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:34:00.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-17T19:34:00.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:34:00.529 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:34:00.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:34:00.630 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:00.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:00.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:00.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:00.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:00.631 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:00.632 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:00.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-17T19:34:00.940 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:00.941 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:00.941 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 6 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:34:01.346 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:34:01.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:34:01.347 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:34:01.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:34:01.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:34:01.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:34:01.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:34:01.756 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:34:01.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:34:01.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:01.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:01.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:01.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:01.958 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:01.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:01.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:02.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:34:02.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:02.284 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:02.284 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:02.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:34:02.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:34:02.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:34:02.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-09-17T19:34:02.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:34:02.670 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-09-17T19:34:02.671 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:34:02.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:34:02.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:34:02.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:34:02.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-17T19:34:02.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:34:02.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:34:02.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:34:02.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:34:02.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:02.984 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-09-17T19:34:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:34:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:34:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:34:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:34:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:34:05.107 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:34:05.108 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:34:05.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:34:05.109 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:34:05.109 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:34:05.109 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:34:05.109 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:34:05.110 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:05.110 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:34:05.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:34:05.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:34:05.112 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:34:05.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:34:05.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:34:05.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:34:05.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:34:05.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:34:05.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:34:05.144 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:05.140+0000 7f58f1bb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:05.147 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:05.144+0000 7f58f1bb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:05.148 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:05.148+0000 7f58f1bb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:05.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:34:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:34:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:34:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:34:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:34:05.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:34:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:34:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:05.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:05.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:06.219 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:06.216+0000 7f58f1bb67c0 -1 Falling back to public interface 2024-09-17T19:34:06.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:06.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:06.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:34:06.803 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:34:06.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:06.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:07.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:08.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:08.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:08.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:34:08.126 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:34:08.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:08.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:08.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:09.150 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:09.148+0000 7f58f1bb67c0 -1 osd.1 97 log_to_monitors true 2024-09-17T19:34:09.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:09.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:09.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:34:09.452 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:34:09.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:09.453 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 101 up_thru 96 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/1413571880,v1:127.0.0.1:6811/1413571880] [v2:127.0.0.1:6812/1413571880,v1:127.0.0.1:6813/1413571880] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:34:09.899 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:34:09.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:34:09.900 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:34:09.900 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:34:09.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:34:09.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:34:09.901 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:34:10.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:34:10.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:34:10.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:34:10.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:34:10.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:34:10.020 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:34:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:34:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:34:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:34:10.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:10.574 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:34:10.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893125 2024-09-17T19:34:10.725 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893125 2024-09-17T19:34:10.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893125' 2024-09-17T19:34:10.726 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:10.726 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:34:10.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696898 2024-09-17T19:34:10.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696898 2024-09-17T19:34:10.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-408021893125 1-433791696898' 2024-09-17T19:34:10.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:10.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-408021893125 2024-09-17T19:34:10.867 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:10.868 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:34:10.869 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-408021893125 2024-09-17T19:34:10.869 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:10.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893125 2024-09-17T19:34:10.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 408021893125' 2024-09-17T19:34:10.870 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 408021893125 2024-09-17T19:34:10.871 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:11.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893124 -lt 408021893125 2024-09-17T19:34:11.181 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:34:12.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:34:12.184 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:12.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893126 -lt 408021893125 2024-09-17T19:34:12.560 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:12.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-433791696898 2024-09-17T19:34:12.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:12.563 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:34:12.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-433791696898 2024-09-17T19:34:12.564 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:12.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696898 2024-09-17T19:34:12.566 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2024-09-17T19:34:12.566 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 433791696898 2024-09-17T19:34:12.566 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:34:12.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696898 -lt 433791696898 2024-09-17T19:34:12.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:34:12.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:12.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:13.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:34:13.307 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:13.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:13.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:13.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:13.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:13.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:13.308 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:13.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:34:13.651 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:13.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:13.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:14.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:34:14.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:34:14.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:34:14.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-09-17T19:34:14.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:34:14.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-09-17T19:34:14.022 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-09-17T19:34:14.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:34:14.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:34:14.023 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-09-17T19:34:14.132 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-17T19:34:14.259 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:34:14.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:34:14.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:34:14.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:34:14.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:34:14.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-17T19:34:14.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:34:14.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:34:14.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:34:14.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:34:14.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:34:14.572 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-09-17T19:34:16.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:34:16.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:34:16.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:34:16.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:34:16.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:34:16.413 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:34:16.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:34:16.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:34:16.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:34:16.414 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:34:16.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:34:16.414 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:34:16.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:34:16.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:34:16.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:34:16.416 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:34:16.447 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:16.444+0000 7f88040fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:16.450 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:16.448+0000 7f88040fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:16.451 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:16.448+0000 7f88040fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:16.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:34:16.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:34:16.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:16.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:34:17.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:17.935 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:17.932+0000 7f88040fc7c0 -1 Falling back to public interface 2024-09-17T19:34:18.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:18.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:18.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:34:18.087 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:34:18.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:18.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:34:18.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:19.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:19.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:19.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:34:19.428 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:34:19.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:19.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:34:19.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:20.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:20.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:20.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:34:20.762 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:34:20.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:20.763 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:34:20.867 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:20.864+0000 7f88040fc7c0 -1 osd.0 102 log_to_monitors true 2024-09-17T19:34:21.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:22.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:22.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:22.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:34:22.113 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:34:22.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:22.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/487454366,v1:127.0.0.1:6803/487454366] [v2:127.0.0.1:6804/487454366,v1:127.0.0.1:6805/487454366] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:34:22.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:34:22.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:34:22.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:34:22.558 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:34:22.558 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:34:22.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:34:22.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:34:22.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:34:22.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:34:22.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:34:22.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:34:22.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:34:22.683 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:34:22.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:34:23.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:34:23.010 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:34:23.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:34:23.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:23.010 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:34:23.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533378 2024-09-17T19:34:23.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533378 2024-09-17T19:34:23.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533378' 2024-09-17T19:34:23.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:23.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:34:23.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=433791696901 2024-09-17T19:34:23.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 433791696901 2024-09-17T19:34:23.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533378 1-433791696901' 2024-09-17T19:34:23.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:23.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533378 2024-09-17T19:34:23.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:23.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:34:23.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533378 2024-09-17T19:34:23.319 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:23.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533378 2024-09-17T19:34:23.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2024-09-17T19:34:23.320 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 455266533378 2024-09-17T19:34:23.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:23.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 455266533378 2024-09-17T19:34:23.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:34:24.624 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:34:24.625 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:24.946 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533378 -lt 455266533378 2024-09-17T19:34:24.947 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:24.947 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-433791696901 2024-09-17T19:34:24.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:24.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:34:24.950 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-433791696901 2024-09-17T19:34:24.950 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:24.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=433791696901 2024-09-17T19:34:24.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 433791696901' 2024-09-17T19:34:24.951 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 433791696901 2024-09-17T19:34:24.952 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:34:25.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 433791696901 -lt 433791696901 2024-09-17T19:34:25.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:34:25.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:25.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:25.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:34:25.663 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:25.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:25.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:25.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:25.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:25.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:25.665 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:26.000 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:34:26.001 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:26.001 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:26.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:26.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:34:26.427 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:34:26.428 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:26.738 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-09-17T19:34:28.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:34:28.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:34:28.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:34:28.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:34:28.957 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:34:28.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:28.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:34:28.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:34:28.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:34:28.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:34:28.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:34:28.993 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:28.992+0000 7fdffb1207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:28.996 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:28.992+0000 7fdffb1207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:28.997 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:28.996+0000 7fdffb1207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:29.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:34:29.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:34:29.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:34:29.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:34:29.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:34:29.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:29.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:34:29.293 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:34:29.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:29.293 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:29.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:30.543 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:30.539+0000 7fdffb1207c0 -1 Falling back to public interface 2024-09-17T19:34:30.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:30.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:30.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:34:30.637 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:34:30.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:30.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:30.964 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:31.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:31.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:31.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:34:31.967 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:34:31.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:31.967 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:32.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:33.285 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:33.283+0000 7fdffb1207c0 -1 osd.1 108 log_to_monitors true 2024-09-17T19:34:33.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:33.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:33.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:34:33.299 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:34:33.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:33.299 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:33.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:34.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:34.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:34.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:34:34.622 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:34:34.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:34.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:34.949 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/808526402,v1:127.0.0.1:6811/808526402] [v2:127.0.0.1:6812/808526402,v1:127.0.0.1:6813/808526402] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:34:34.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:34:34.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:34:34.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:34:34.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:34:34.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:34:34.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:34:34.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:34:34.950 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:34:34.951 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:34:34.951 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:34:34.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:34:34.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:34:34.951 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:34:35.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:34:35.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:34:35.059 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:34:35.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:34:35.060 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:34:35.060 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:34:35.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:34:35.384 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:34:35.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:34:35.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:35.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:34:35.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533381 2024-09-17T19:34:35.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533381 2024-09-17T19:34:35.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533381' 2024-09-17T19:34:35.551 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:35.551 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:34:35.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=481036337154 2024-09-17T19:34:35.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 481036337154 2024-09-17T19:34:35.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533381 1-481036337154' 2024-09-17T19:34:35.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:35.693 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533381 2024-09-17T19:34:35.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:35.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:34:35.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533381 2024-09-17T19:34:35.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:35.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533381 2024-09-17T19:34:35.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533381' 2024-09-17T19:34:35.696 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 455266533381 2024-09-17T19:34:35.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:36.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533380 -lt 455266533381 2024-09-17T19:34:36.024 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:34:37.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:34:37.026 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:37.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533381 -lt 455266533381 2024-09-17T19:34:37.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:37.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-481036337154 2024-09-17T19:34:37.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:37.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:34:37.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-481036337154 2024-09-17T19:34:37.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=481036337154 2024-09-17T19:34:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-09-17T19:34:37.367 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 481036337154 2024-09-17T19:34:37.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:34:37.706 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 481036337154 -lt 481036337154 2024-09-17T19:34:37.707 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:34:37.708 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:37.708 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:38.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:34:38.104 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:38.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:38.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:38.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:38.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:38.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:38.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:38.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:34:38.415 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:38.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:38.416 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-17T19:34:38.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:34:38.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:39.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-09-17T19:34:41.440 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:34:41.441 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:34:41.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:34:41.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:34:41.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:34:41.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:34:41.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:34:41.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:34:41.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:34:41.445 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:34:41.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:34:41.445 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:34:41.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:34:41.447 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:34:41.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:34:41.448 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:34:41.479 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:41.475+0000 7f767f6aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:41.586 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:41.479+0000 7f767f6aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:41.586 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:41.479+0000 7f767f6aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:41.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:42.150 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:42.547 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:42.543+0000 7f767f6aa7c0 -1 Falling back to public interface 2024-09-17T19:34:43.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:43.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:43.151 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:34:43.151 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:34:43.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:43.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:43.483 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:44.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:44.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:44.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:34:44.485 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:34:44.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:44.486 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:44.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:45.232 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:45.227+0000 7f767f6aa7c0 -1 osd.1 113 log_to_monitors true 2024-09-17T19:34:45.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:45.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:45.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:34:45.833 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:34:45.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:45.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:46.165 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 117 up_thru 112 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/3150985260,v1:127.0.0.1:6811/3150985260] [v2:127.0.0.1:6812/3150985260,v1:127.0.0.1:6813/3150985260] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:34:46.166 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:34:46.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:34:46.167 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:34:46.167 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:34:46.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:34:46.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:34:46.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:34:46.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:34:46.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:34:46.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:34:46.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:34:46.274 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:34:46.274 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:34:46.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:34:46.873 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:34:46.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:34:46.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:46.874 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:34:47.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533385 2024-09-17T19:34:47.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533385 2024-09-17T19:34:47.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533385' 2024-09-17T19:34:47.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:34:47.099 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:34:47.305 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=502511173634 2024-09-17T19:34:47.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 502511173634 2024-09-17T19:34:47.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533385 1-502511173634' 2024-09-17T19:34:47.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:47.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533385 2024-09-17T19:34:47.307 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:47.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:34:47.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533385 2024-09-17T19:34:47.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:47.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533385 2024-09-17T19:34:47.311 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533385' 2024-09-17T19:34:47.311 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 455266533385 2024-09-17T19:34:47.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:47.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533383 -lt 455266533385 2024-09-17T19:34:47.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:34:48.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:34:48.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:34:48.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533385 -lt 455266533385 2024-09-17T19:34:48.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:34:48.950 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-502511173634 2024-09-17T19:34:48.950 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:34:48.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:34:48.952 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-502511173634 2024-09-17T19:34:48.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:34:48.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=502511173634 2024-09-17T19:34:48.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-09-17T19:34:48.954 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 502511173634 2024-09-17T19:34:48.954 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:34:49.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 502511173634 -lt 502511173634 2024-09-17T19:34:49.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:34:49.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:49.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:49.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:34:49.710 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:49.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:49.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:49.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:49.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:49.711 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:49.712 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:50.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:34:50.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:50.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:50.047 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:50.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:34:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:34:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:34:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:34:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:34:50.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:34:50.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:34:50.534 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:50.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:50.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:50.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:50.535 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:50.536 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:50.536 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:50.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:34:50.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:50.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:50.843 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:34:51.239 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:34:51.240 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:34:51.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:34:51.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:34:51.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:34:51.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:34:51.636 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:34:51.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:34:51.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:34:51.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:34:51.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:34:51.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:34:51.838 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:34:51.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:34:51.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:34:52.146 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:34:52.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:34:52.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:34:52.147 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:34:52.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:34:52.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:34:52.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:34:52.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-09-17T19:34:52.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:34:52.546 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-17T19:34:52.546 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:34:52.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:34:52.549 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:34:52.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:34:52.550 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:34:52.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:52.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-09-17T19:34:55.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:34:55.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:34:55.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:34:55.380 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:34:55.381 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:34:55.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:34:55.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:34:55.381 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:34:55.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:34:55.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:34:55.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:34:55.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:34:55.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:34:55.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:34:55.386 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:34:55.417 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:55.415+0000 7fd96833d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:55.420 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:55.419+0000 7fd96833d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:55.422 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:55.419+0000 7fd96833d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:34:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:34:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:34:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:34:55.722 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:34:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:34:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:34:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:34:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:55.723 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:56.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:56.483 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:56.479+0000 7fd96833d7c0 -1 Falling back to public interface 2024-09-17T19:34:57.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:57.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:57.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:34:57.068 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:34:57.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:57.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:57.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:34:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:34:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:58.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:34:58.715 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:34:59.126 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:34:59.123+0000 7fd96833d7c0 -1 osd.1 119 log_to_monitors true 2024-09-17T19:34:59.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:34:59.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:34:59.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:34:59.717 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:34:59.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:34:59.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 123 up_thru 117 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/1548873428,v1:127.0.0.1:6811/1548873428] [v2:127.0.0.1:6812/1548873428,v1:127.0.0.1:6813/1548873428] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:35:00.281 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:35:00.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:35:00.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:35:00.282 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:35:00.282 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:35:00.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:35:00.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:35:00.283 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:35:00.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:35:00.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:35:00.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:35:00.403 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:35:00.404 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:35:00.404 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:35:00.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:35:00.757 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:35:00.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:35:00.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:00.757 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:35:00.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533388 2024-09-17T19:35:00.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533388 2024-09-17T19:35:00.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533388' 2024-09-17T19:35:00.948 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:00.948 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:35:01.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=528280977410 2024-09-17T19:35:01.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 528280977410 2024-09-17T19:35:01.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533388 1-528280977410' 2024-09-17T19:35:01.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:01.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533388 2024-09-17T19:35:01.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:01.101 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:35:01.101 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533388 2024-09-17T19:35:01.101 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:01.102 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533388 2024-09-17T19:35:01.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533388' 2024-09-17T19:35:01.103 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 455266533388 2024-09-17T19:35:01.103 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:01.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533387 -lt 455266533388 2024-09-17T19:35:01.422 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:35:02.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:35:02.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:02.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533389 -lt 455266533388 2024-09-17T19:35:02.749 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:02.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-528280977410 2024-09-17T19:35:02.749 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:02.751 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:35:02.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-528280977410 2024-09-17T19:35:02.752 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:02.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=528280977410 2024-09-17T19:35:02.754 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-09-17T19:35:02.754 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 528280977410 2024-09-17T19:35:02.754 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:35:03.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 528280977410 -lt 528280977410 2024-09-17T19:35:03.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:35:03.112 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:03.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:03.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:35:03.513 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:03.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:03.513 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:03.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:03.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:03.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:03.514 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:03.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:35:03.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:03.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:03.873 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:04.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:35:04.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:35:04.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:35:04.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:35:04.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:04.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:35:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:04.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:04.388 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:04.389 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:04.701 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:35:04.702 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:04.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:04.703 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' 4 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:35:05.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:35:05.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:35:05.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:35:05.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:35:05.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:35:05.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:35:05.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:05.461 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:35:05.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:05.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:05.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:05.663 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:05.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:05.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:05.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:05.664 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:05.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:35:05.988 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:05.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:05.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:06.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:35:06.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:35:06.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:35:06.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-09-17T19:35:06.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:35:06.493 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:35:06.494 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:35:06.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:35:06.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:35:06.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:35:06.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:35:06.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:35:06.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:35:06.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:35:06.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:35:09.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:35:09.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:35:09.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:35:09.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:35:09.095 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:35:09.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:35:09.097 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:35:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:35:09.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:35:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:35:09.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:35:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:35:09.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:35:09.131 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:09.127+0000 7ff1080dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:09.133 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:09.131+0000 7ff1080dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:09.135 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:09.131+0000 7ff1080dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:09.431 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:09.432 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:09.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:10.395 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:10.391+0000 7ff1080dc7c0 -1 Falling back to public interface 2024-09-17T19:35:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:35:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:35:10.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:10.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:11.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:12.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:12.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:12.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:35:12.114 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:35:12.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:12.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:12.443 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:13.260 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:13.259+0000 7ff1080dc7c0 -1 osd.1 125 log_to_monitors true 2024-09-17T19:35:13.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:13.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:13.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:35:13.445 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:35:13.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:13.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:13.845 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 129 up_thru 123 down_at 126 last_clean_interval [123,125) [v2:127.0.0.1:6810/2154050175,v1:127.0.0.1:6811/2154050175] [v2:127.0.0.1:6812/2154050175,v1:127.0.0.1:6813/2154050175] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:35:13.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:35:13.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:35:13.847 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:35:13.848 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:35:13.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:35:13.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:35:13.848 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:35:13.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:35:13.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:35:13.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:35:13.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:35:13.959 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:35:13.959 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:35:14.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:35:14.292 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:35:14.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:35:14.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:14.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:35:14.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=455266533392 2024-09-17T19:35:14.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 455266533392 2024-09-17T19:35:14.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533392' 2024-09-17T19:35:14.442 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:14.442 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:35:14.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=554050781186 2024-09-17T19:35:14.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 554050781186 2024-09-17T19:35:14.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-455266533392 1-554050781186' 2024-09-17T19:35:14.629 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:14.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-455266533392 2024-09-17T19:35:14.630 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:14.632 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:35:14.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-455266533392 2024-09-17T19:35:14.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:14.634 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=455266533392 2024-09-17T19:35:14.635 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 455266533392' 2024-09-17T19:35:14.635 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 455266533392 2024-09-17T19:35:14.635 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:14.986 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533391 -lt 455266533392 2024-09-17T19:35:14.987 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:35:15.988 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:35:15.989 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:16.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 455266533392 -lt 455266533392 2024-09-17T19:35:16.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:16.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-554050781186 2024-09-17T19:35:16.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:16.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:35:16.320 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-554050781186 2024-09-17T19:35:16.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:16.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=554050781186 2024-09-17T19:35:16.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 554050781186' 2024-09-17T19:35:16.322 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 554050781186 2024-09-17T19:35:16.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:35:16.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 554050781186 -lt 554050781186 2024-09-17T19:35:16.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:35:16.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:16.649 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:17.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:35:17.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:17.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:17.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:17.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:17.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:17.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:17.067 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:17.402 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:35:17.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:17.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:17.403 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:17.788 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:35:17.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:35:17.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:35:17.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:35:17.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:17.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:35:17.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:17.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:17.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:17.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:17.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:17.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:17.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:17.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:18.212 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:35:18.213 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:18.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:18.214 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:18.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:35:18.577 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:35:18.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:35:18.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-17T19:35:18.579 INFO:tasks.workunit.client.0.smithi105.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-17T19:35:18.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:18.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-17T19:35:18.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-09-17T19:35:18.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:35:18.580 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:18.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:18.581 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-09-17T19:35:18.581 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-09-17T19:35:18.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:35:18.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:18.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:18.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-09-17T19:35:18.582 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-09-17T19:35:18.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:35:18.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:18.583 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:18.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-09-17T19:35:18.584 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-09-17T19:35:18.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:35:18.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:18.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:18.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-09-17T19:35:18.586 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-09-17T19:35:18.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:35:18.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:18.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-17T19:35:18.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-17T19:35:18.588 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:35:18.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:35:18.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:35:18.590 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:35:18.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:35:18.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:35:18.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:35:18.895 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:35:18.896 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-09-17T19:35:21.122 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:35:21.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:35:21.124 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:35:21.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:35:21.126 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:35:21.127 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:35:21.128 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:35:21.158 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:21.154+0000 7f11ee8107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:21.161 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:21.158+0000 7f11ee8107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:21.162 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:21.158+0000 7f11ee8107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:21.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:35:21.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:35:21.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:35:21.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:35:21.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:35:21.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:21.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:35:21.459 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:35:21.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:21.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:35:21.815 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:22.208 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:22.202+0000 7f11ee8107c0 -1 Falling back to public interface 2024-09-17T19:35:22.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:22.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:22.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:35:22.817 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:35:22.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:22.818 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:35:23.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:24.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:24.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:24.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:35:24.154 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:35:24.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:24.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:35:24.478 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:25.059 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:25.054+0000 7f11ee8107c0 -1 osd.0 131 log_to_monitors true 2024-09-17T19:35:25.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:25.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:25.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:35:25.481 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:35:25.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:25.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:35:25.888 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 135 up_thru 129 down_at 132 last_clean_interval [106,131) [v2:127.0.0.1:6802/1809199627,v1:127.0.0.1:6803/1809199627] [v2:127.0.0.1:6804/1809199627,v1:127.0.0.1:6805/1809199627] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:35:25.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:35:25.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:35:25.891 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:35:25.891 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:35:25.891 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:35:25.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:35:25.892 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:35:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:35:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:35:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:35:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:35:26.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:35:26.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:35:26.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:35:26.394 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:35:26.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:35:26.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:26.395 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:35:26.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=579820584962 2024-09-17T19:35:26.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 579820584962 2024-09-17T19:35:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-579820584962' 2024-09-17T19:35:26.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:26.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:35:26.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=554050781189 2024-09-17T19:35:26.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 554050781189 2024-09-17T19:35:26.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-579820584962 1-554050781189' 2024-09-17T19:35:26.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:26.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-579820584962 2024-09-17T19:35:26.796 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:35:26.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-579820584962 2024-09-17T19:35:26.798 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:26.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=579820584962 2024-09-17T19:35:26.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 579820584962' 2024-09-17T19:35:26.799 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 579820584962 2024-09-17T19:35:26.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:27.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 579820584962 2024-09-17T19:35:27.115 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:35:28.117 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:35:28.118 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:28.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 579820584962 -lt 579820584962 2024-09-17T19:35:28.454 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:28.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:28.455 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-554050781189 2024-09-17T19:35:28.457 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:35:28.458 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-554050781189 2024-09-17T19:35:28.458 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:28.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=554050781189 2024-09-17T19:35:28.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 554050781189' 2024-09-17T19:35:28.460 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 554050781189 2024-09-17T19:35:28.460 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:35:28.791 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 554050781189 -lt 554050781189 2024-09-17T19:35:28.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:35:28.792 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:28.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:29.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:35:29.201 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:29.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:29.202 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:29.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:29.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:29.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:29.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:29.540 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:35:29.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:29.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:29.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:29.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:35:29.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:35:29.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:35:29.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:35:29.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:29.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:35:30.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:30.016 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:30.017 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:30.017 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:30.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:30.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:30.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:30.018 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:30.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:35:30.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:30.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:30.322 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:30.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:35:30.719 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:35:30.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:35:30.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:35:30.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:35:30.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:35:30.720 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:35:30.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:35:30.721 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:35:31.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:35:31.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:35:31.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:35:31.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:31.121 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:35:31.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:31.324 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:31.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:31.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:31.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:31.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:31.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:31.325 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:31.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:35:31.653 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:31.653 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:31.654 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:32.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:35:32.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:35:32.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:35:32.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:35:32.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:35:32.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:35:32.050 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:35:32.050 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:35:32.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:35:32.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:35:32.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:35:32.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 13 )) 2024-09-17T19:35:32.463 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:32.464 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-17T19:35:32.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:32.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:32.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:32.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:32.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:32.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:32.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:32.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:33.196 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:35:33.196 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:33.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:33.197 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-17T19:35:33.595 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:35:33.596 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:35:33.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:35:35.825 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:35:35.826 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:35:35.828 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:35:35.828 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:35:35.828 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:35:35.828 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:35.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:35:35.829 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:35:35.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:35:35.830 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:35:35.830 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:35:35.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:35:35.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:35:35.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:35:35.863 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:35.858+0000 7ff35db967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:35.866 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:35.862+0000 7ff35db967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:35.867 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:35.862+0000 7ff35db967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:36.183 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:36.523 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:37.159 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:37.154+0000 7ff35db967c0 -1 Falling back to public interface 2024-09-17T19:35:37.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:37.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:37.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:35:37.525 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:35:37.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:37.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:37.836 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:38.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:38.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:38.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:35:38.838 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:35:38.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:38.838 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:39.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:39.873 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:39.870+0000 7ff35db967c0 -1 osd.1 136 log_to_monitors true 2024-09-17T19:35:40.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:40.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:40.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:35:40.155 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:35:40.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:40.156 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:40.621 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 140 up_thru 135 down_at 137 last_clean_interval [129,136) [v2:127.0.0.1:6810/1645007662,v1:127.0.0.1:6811/1645007662] [v2:127.0.0.1:6812/1645007662,v1:127.0.0.1:6813/1645007662] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:35:40.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:35:40.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:35:40.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:35:40.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:35:40.621 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:35:40.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:35:40.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:35:40.622 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:35:40.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:35:40.623 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:35:40.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:35:40.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:35:40.624 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:35:40.747 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:35:40.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:35:40.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:35:40.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:35:40.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:35:40.748 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:35:41.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:35:41.167 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:35:41.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:35:41.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:41.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:35:41.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=579820584966 2024-09-17T19:35:41.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 579820584966 2024-09-17T19:35:41.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-579820584966' 2024-09-17T19:35:41.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:41.385 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:35:41.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=601295421442 2024-09-17T19:35:41.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 601295421442 2024-09-17T19:35:41.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-579820584966 1-601295421442' 2024-09-17T19:35:41.552 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:41.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-579820584966 2024-09-17T19:35:41.554 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:41.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:35:41.556 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-579820584966 2024-09-17T19:35:41.556 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:41.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=579820584966 2024-09-17T19:35:41.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 579820584966' 2024-09-17T19:35:41.557 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 579820584966 2024-09-17T19:35:41.558 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:41.887 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 579820584964 -lt 579820584966 2024-09-17T19:35:41.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:35:42.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:35:42.890 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:43.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 579820584966 -lt 579820584966 2024-09-17T19:35:43.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:43.235 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-601295421442 2024-09-17T19:35:43.235 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:43.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:35:43.237 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-601295421442 2024-09-17T19:35:43.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:43.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=601295421442 2024-09-17T19:35:43.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 601295421442' 2024-09-17T19:35:43.239 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 601295421442 2024-09-17T19:35:43.240 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:35:43.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 601295421442 -lt 601295421442 2024-09-17T19:35:43.569 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:35:43.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:43.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:43.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:35:43.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:43.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:43.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:43.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:43.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:43.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:43.982 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:44.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:35:44.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:44.293 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:44.293 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:44.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:35:44.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:35:44.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:35:44.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:35:44.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:44.699 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:35:44.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:44.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:44.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:44.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:44.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:44.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:44.802 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:44.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:45.129 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:35:45.130 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:45.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:45.131 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:35:45.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:35:45.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:35:45.525 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:35:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:35:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:35:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:35:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:35:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:35:46.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:35:46.136 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:46.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:46.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:46.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:46.138 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:46.139 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:46.139 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:46.451 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:35:46.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:46.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:46.452 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:46.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:35:46.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:35:46.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:35:46.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-09-17T19:35:46.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:46.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-09-17T19:35:46.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-09-17T19:35:46.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:35:46.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:46.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-17T19:35:46.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:35:46.844 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:35:46.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:35:47.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:35:47.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-09-17T19:35:47.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:35:47.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:35:47.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:35:47.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:35:47.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:35:47.155 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-09-17T19:35:48.743 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:35:49.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:35:49.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:35:49.280 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:35:49.282 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:35:49.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:35:49.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:35:49.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:35:49.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:35:49.283 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:35:49.283 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:35:49.284 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:35:49.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:35:49.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:35:49.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:35:49.284 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:35:49.316 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:49.314+0000 7fb6d5cd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:49.319 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:49.314+0000 7fb6d5cd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:49.320 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:49.318+0000 7fb6d5cd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:49.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:49.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:50.599 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:50.594+0000 7fb6d5cd77c0 -1 Falling back to public interface 2024-09-17T19:35:50.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:50.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:50.961 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:35:50.961 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:35:50.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:50.962 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:51.295 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:52.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:52.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:52.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:35:52.298 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:35:52.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:52.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:52.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:35:53.299 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:35:53.294+0000 7fb6d5cd77c0 -1 osd.1 142 log_to_monitors true 2024-09-17T19:35:53.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:35:53.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:35:53.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:35:53.640 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:35:53.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:35:53.640 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:35:54.077 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 146 up_thru 140 down_at 143 last_clean_interval [140,142) [v2:127.0.0.1:6810/3531890150,v1:127.0.0.1:6811/3531890150] [v2:127.0.0.1:6812/3531890150,v1:127.0.0.1:6813/3531890150] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:35:54.078 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:35:54.079 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:35:54.079 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:35:54.079 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:35:54.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:35:54.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:35:54.080 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:35:54.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:35:54.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:35:54.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:35:54.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:35:54.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:35:54.193 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:35:54.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:35:54.658 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:35:54.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:35:54.658 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:54.658 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:35:54.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=579820584969 2024-09-17T19:35:54.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 579820584969 2024-09-17T19:35:54.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-579820584969' 2024-09-17T19:35:54.834 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:35:54.834 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:35:55.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=627065225218 2024-09-17T19:35:55.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 627065225218 2024-09-17T19:35:55.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-579820584969 1-627065225218' 2024-09-17T19:35:55.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:55.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-579820584969 2024-09-17T19:35:55.083 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:55.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:35:55.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-579820584969 2024-09-17T19:35:55.086 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:55.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=579820584969 2024-09-17T19:35:55.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 579820584969' 2024-09-17T19:35:55.088 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 579820584969 2024-09-17T19:35:55.088 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:55.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 579820584968 -lt 579820584969 2024-09-17T19:35:55.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:35:56.424 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:35:56.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:35:56.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 579820584970 -lt 579820584969 2024-09-17T19:35:56.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:35:56.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-627065225218 2024-09-17T19:35:56.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:35:56.762 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:35:56.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-627065225218 2024-09-17T19:35:56.763 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:35:56.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=627065225218 2024-09-17T19:35:56.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 627065225218' 2024-09-17T19:35:56.764 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 627065225218 2024-09-17T19:35:56.765 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:35:57.103 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 627065225218 -lt 627065225218 2024-09-17T19:35:57.104 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:35:57.104 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:57.105 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:57.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:35:57.515 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:35:57.516 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:35:57.516 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:35:57.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:35:57.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:35:57.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:35:57.517 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:35:57.840 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:35:57.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:35:57.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:35:57.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:35:58.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:35:58.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:35:58.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:35:58.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:35:58.224 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-09-17T19:35:58.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-09-17T19:35:58.226 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:35:58.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:35:58.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:35:58.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:35:58.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-09-17T19:35:58.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:35:58.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:35:58.536 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:35:58.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:35:58.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:35:58.537 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-09-17T19:36:00.010 INFO:tasks.workunit.client.0.smithi105.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-17T19:36:00.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:36:00.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:36:00.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:36:00.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:36:00.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:36:00.796 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:36:00.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:36:00.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:36:00.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:36:00.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:36:00.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:36:00.800 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:36:00.800 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:36:00.800 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:36:00.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:36:00.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:36:00.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:36:00.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:36:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:36:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:36:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:36:00.803 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:36:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:36:00.803 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:36:00.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:36:00.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:36:00.836 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:00.834+0000 7f4d912837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:00.838 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:00.838+0000 7f4d912837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:00.840 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:00.842+0000 7f4d912837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:01.134 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:36:01.135 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:36:01.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:01.135 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:01.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:02.135 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:02.134+0000 7f4d912837c0 -1 Falling back to public interface 2024-09-17T19:36:02.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:02.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:02.471 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:36:02.471 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:36:02.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:02.472 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:02.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:03.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:03.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:03.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:36:03.808 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:36:03.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:03.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:04.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:04.841 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:04.842+0000 7f4d912837c0 -1 osd.0 148 log_to_monitors true 2024-09-17T19:36:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:36:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:36:05.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:05.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:05.591 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 152 up_thru 152 down_at 149 last_clean_interval [135,148) [v2:127.0.0.1:6802/2489956495,v1:127.0.0.1:6803/2489956495] [v2:127.0.0.1:6804/2489956495,v1:127.0.0.1:6805/2489956495] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:36:05.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:36:05.591 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:36:05.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:36:05.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:36:05.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:36:05.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:36:05.592 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:36:05.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:36:05.593 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:36:05.593 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:36:05.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:36:05.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:36:05.593 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:36:05.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:36:05.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:36:05.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:36:05.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:36:05.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:36:05.717 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:36:06.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:36:06.169 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:36:06.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:36:06.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:06.170 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:36:06.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=652835028994 2024-09-17T19:36:06.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 652835028994 2024-09-17T19:36:06.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-652835028994' 2024-09-17T19:36:06.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:06.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:36:06.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=627065225221 2024-09-17T19:36:06.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 627065225221 2024-09-17T19:36:06.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-652835028994 1-627065225221' 2024-09-17T19:36:06.541 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:06.542 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-652835028994 2024-09-17T19:36:06.542 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:06.543 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:36:06.543 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-652835028994 2024-09-17T19:36:06.544 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:06.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=652835028994 2024-09-17T19:36:06.545 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 652835028994' 2024-09-17T19:36:06.545 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 652835028994 2024-09-17T19:36:06.546 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:06.872 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 652835028994 2024-09-17T19:36:06.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:36:07.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:36:07.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:08.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 652835028994 -lt 652835028994 2024-09-17T19:36:08.237 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:08.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-627065225221 2024-09-17T19:36:08.238 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:08.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:36:08.242 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-627065225221 2024-09-17T19:36:08.242 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:08.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=627065225221 2024-09-17T19:36:08.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 627065225221' 2024-09-17T19:36:08.243 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 627065225221 2024-09-17T19:36:08.244 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:08.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 627065225221 -lt 627065225221 2024-09-17T19:36:08.578 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:36:08.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:08.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:09.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:36:09.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:36:09.001 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:36:09.001 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:36:09.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:36:09.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:36:09.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:36:09.002 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:36:09.355 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=4 2024-09-17T19:36:09.356 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:36:09.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:09.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 = 6 2024-09-17T19:36:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 4 '!=' -1 2024-09-17T19:36:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:36:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=4 2024-09-17T19:36:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:36:09.744 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:36:09.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:36:09.845 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:36:09.846 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:36:09.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:36:09.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:36:09.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:36:09.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:36:09.847 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:36:10.153 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:10.154 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:10.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:36:10.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:36:10.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:36:10.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:36:10.532 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:36:10.533 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:36:10.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:36:10.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-17T19:36:10.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:36:10.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:36:10.842 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:36:10.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:36:10.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:36:10.843 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-09-17T19:36:12.707 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-17T19:36:12.707 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:36:13.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:36:13.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:13.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:13.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:36:13.247 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:36:13.248 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:36:13.249 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:36:13.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:36:13.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:36:13.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:36:13.249 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:36:13.281 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:13.281+0000 7fcce95a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:13.283 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:13.281+0000 7fcce95a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:13.285 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:13.285+0000 7fcce95a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:13.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:36:13.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:36:13.574 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:13.575 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:13.904 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:14.587 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:14.585+0000 7fcce95a27c0 -1 Falling back to public interface 2024-09-17T19:36:14.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:14.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:14.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:36:14.907 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:36:14.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:14.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:15.231 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:16.233 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:16.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:16.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:36:16.234 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:36:16.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:16.235 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:16.557 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:17.546 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:17.545+0000 7fcce95a27c0 -1 osd.1 154 log_to_monitors true 2024-09-17T19:36:17.558 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:17.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:17.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:36:17.559 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:36:17.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:17.559 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:17.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:18.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:18.909 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:18.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:36:18.910 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:36:18.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:18.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:19.244 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 158 up_thru 158 down_at 155 last_clean_interval [146,154) [v2:127.0.0.1:6810/228094118,v1:127.0.0.1:6811/228094118] [v2:127.0.0.1:6812/228094118,v1:127.0.0.1:6813/228094118] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:36:19.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:36:19.244 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:36:19.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:36:19.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:36:19.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:36:19.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:36:19.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:36:19.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:36:19.246 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:36:19.246 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:36:19.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:36:19.246 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:36:19.247 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:36:19.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:36:19.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:36:19.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:36:19.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:36:19.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:36:19.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:36:19.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:36:19.688 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:36:19.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:36:19.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:19.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:36:19.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=652835028997 2024-09-17T19:36:19.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 652835028997 2024-09-17T19:36:19.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-652835028997' 2024-09-17T19:36:19.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:19.866 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:36:20.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=678604832770 2024-09-17T19:36:20.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 678604832770 2024-09-17T19:36:20.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-652835028997 1-678604832770' 2024-09-17T19:36:20.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:20.011 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-652835028997 2024-09-17T19:36:20.011 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:20.011 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:36:20.012 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-652835028997 2024-09-17T19:36:20.012 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:20.013 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=652835028997 2024-09-17T19:36:20.014 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 652835028997' 2024-09-17T19:36:20.014 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 652835028997 2024-09-17T19:36:20.014 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:20.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 652835028997 -lt 652835028997 2024-09-17T19:36:20.329 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:20.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-678604832770 2024-09-17T19:36:20.330 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:20.331 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:36:20.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-678604832770 2024-09-17T19:36:20.332 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:20.333 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=678604832770 2024-09-17T19:36:20.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 678604832770' 2024-09-17T19:36:20.334 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 678604832770 2024-09-17T19:36:20.334 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:20.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 678604832770 2024-09-17T19:36:20.638 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:36:21.639 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:36:21.640 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 678604832770 2024-09-17T19:36:21.963 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:36:22.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:36:22.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:23.292 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 678604832770 -lt 678604832770 2024-09-17T19:36:23.293 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:36:23.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:23.294 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:23.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:36:23.688 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:36:23.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:36:23.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:36:23.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:36:23.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:36:23.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:36:23.690 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:36:24.010 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:36:24.011 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:36:24.011 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:24.011 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:24.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:36:24.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:36:24.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:36:24.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:36:24.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-09-17T19:36:24.379 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:36:24.382 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:36:24.383 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:36:25.695 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:27.479 INFO:tasks.workunit.client.0.smithi105.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:36:28.091 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:36:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:36:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:36:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:36:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:36:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:36:28.092 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:36:28.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:36:28.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:36:28.093 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:36:28.093 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:36:28.093 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:36:28.093 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:36:28.094 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:36:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:36:28.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:36:28.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:36:28.096 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:36:28.097 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:36:28.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:36:28.098 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:36:28.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:36:28.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:36:28.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:36:28.132 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:28.133+0000 7f72761f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:28.135 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:28.133+0000 7f72761f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:28.136 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:28.137+0000 7f72761f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:36:28.429 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:36:28.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:28.430 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:28.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:29.207 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:29.205+0000 7f72761f57c0 -1 Falling back to public interface 2024-09-17T19:36:29.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:29.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:29.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:36:29.761 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:36:29.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:29.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:30.093 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:31.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:31.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:31.095 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:36:31.095 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:36:31.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:31.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:31.433 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:32.082 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:32.081+0000 7f72761f57c0 -1 osd.0 160 log_to_monitors true 2024-09-17T19:36:32.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:32.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:32.434 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:36:32.435 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:36:32.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:32.435 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:32.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:33.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:33.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:33.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:36:33.772 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:36:33.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:33.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:34.099 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 164 up_thru 164 down_at 161 last_clean_interval [152,160) [v2:127.0.0.1:6802/1053938622,v1:127.0.0.1:6803/1053938622] [v2:127.0.0.1:6804/1053938622,v1:127.0.0.1:6805/1053938622] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:36:34.100 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:36:34.101 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:36:34.101 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:36:34.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:36:34.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:36:34.102 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:36:34.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:36:34.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:36:34.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:36:34.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:36:34.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:36:34.206 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:36:34.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:36:34.616 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:36:34.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:36:34.617 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:34.617 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:36:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=704374636546 2024-09-17T19:36:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 704374636546 2024-09-17T19:36:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-704374636546' 2024-09-17T19:36:34.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:34.771 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:36:34.910 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=678604832774 2024-09-17T19:36:34.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 678604832774 2024-09-17T19:36:34.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-704374636546 1-678604832774' 2024-09-17T19:36:34.911 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:34.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-704374636546 2024-09-17T19:36:34.911 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:34.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:36:34.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-704374636546 2024-09-17T19:36:34.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:34.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=704374636546 2024-09-17T19:36:34.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 704374636546' 2024-09-17T19:36:34.914 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 704374636546 2024-09-17T19:36:34.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:35.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 704374636546 2024-09-17T19:36:35.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:36:36.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:36:36.229 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:36.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 704374636546 -lt 704374636546 2024-09-17T19:36:36.610 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:36.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-678604832774 2024-09-17T19:36:36.611 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:36.613 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:36:36.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-678604832774 2024-09-17T19:36:36.614 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:36.615 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=678604832774 2024-09-17T19:36:36.616 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 678604832774' 2024-09-17T19:36:36.616 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 678604832774 2024-09-17T19:36:36.616 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:36.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 678604832774 -lt 678604832774 2024-09-17T19:36:36.954 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:36:36.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:36.955 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:37.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:36:37.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:36:37.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:36:37.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:36:37.344 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:36:37.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:36:37.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:36:37.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:36:37.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:36:37.666 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:36:37.667 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:37.667 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:36:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:36:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:36:38.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:36:38.062 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:36:38.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:36:38.063 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:36:38.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:36:38.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:36:38.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:36:38.370 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:36:38.371 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-09-17T19:36:39.479 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-17T19:36:39.479 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:36:40.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:36:40.048 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:36:40.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:36:40.050 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:36:40.051 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:36:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:36:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:36:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:36:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:36:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:36:40.052 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:36:40.053 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:36:40.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:36:40.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:36:40.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:36:40.053 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:36:40.085 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:40.085+0000 7f87d98087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:40.087 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:40.085+0000 7f87d98087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:40.089 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:40.089+0000 7f87d98087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:40.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:36:40.384 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:40.385 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:40.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:41.139 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:41.137+0000 7f87d98087c0 -1 Falling back to public interface 2024-09-17T19:36:41.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:41.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:41.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:36:41.736 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:36:41.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:41.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:42.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:43.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:43.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:43.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:36:43.100 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:36:43.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:43.100 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:43.450 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:43.662 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:43.661+0000 7f87d98087c0 -1 osd.1 166 log_to_monitors true 2024-09-17T19:36:44.452 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:44.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:44.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:36:44.554 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:36:44.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:44.554 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:36:44.884 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 170 up_thru 170 down_at 167 last_clean_interval [158,166) [v2:127.0.0.1:6810/2813045018,v1:127.0.0.1:6811/2813045018] [v2:127.0.0.1:6812/2813045018,v1:127.0.0.1:6813/2813045018] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:36:44.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:36:44.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:36:44.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:36:44.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:36:44.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:36:44.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:36:44.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:36:44.885 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:36:44.885 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:36:44.885 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:36:44.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:36:44.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:36:44.886 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:36:45.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:36:45.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:36:45.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:36:45.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:36:45.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:36:45.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:36:45.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:36:45.338 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:36:45.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:36:45.338 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:45.338 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:36:45.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=704374636549 2024-09-17T19:36:45.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 704374636549 2024-09-17T19:36:45.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-704374636549' 2024-09-17T19:36:45.500 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:45.501 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:36:45.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=730144440322 2024-09-17T19:36:45.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 730144440322 2024-09-17T19:36:45.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-704374636549 1-730144440322' 2024-09-17T19:36:45.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:45.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-704374636549 2024-09-17T19:36:45.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:45.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:36:45.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-704374636549 2024-09-17T19:36:45.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:45.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=704374636549 2024-09-17T19:36:45.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 704374636549' 2024-09-17T19:36:45.651 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 704374636549 2024-09-17T19:36:45.651 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:45.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 704374636548 -lt 704374636549 2024-09-17T19:36:45.973 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:36:46.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:36:46.975 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:47.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 704374636549 -lt 704374636549 2024-09-17T19:36:47.306 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:47.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:47.306 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-730144440322 2024-09-17T19:36:47.308 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:36:47.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-730144440322 2024-09-17T19:36:47.309 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:47.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=730144440322 2024-09-17T19:36:47.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 730144440322' 2024-09-17T19:36:47.311 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 730144440322 2024-09-17T19:36:47.311 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:47.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 730144440322 -lt 730144440322 2024-09-17T19:36:47.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:36:47.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:47.678 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:48.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:36:48.096 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:36:48.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:36:48.097 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:36:48.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:36:48.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:36:48.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:36:48.098 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:36:48.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:36:48.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:36:48.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:48.410 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:48.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:36:48.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:36:48.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:36:48.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:36:48.770 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-09-17T19:36:48.770 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-09-17T19:36:48.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-09-17T19:36:48.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:36:48.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-09-17T19:36:48.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-09-17T19:36:48.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:36:48.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:36:48.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:36:48.775 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:36:48.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:36:48.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:36:48.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:36:49.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:36:49.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:36:49.082 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:36:49.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:36:49.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:36:49.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:36:49.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:36:49.083 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:36:50.651 INFO:tasks.workunit.client.0.smithi105.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:36:51.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:36:51.313 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:36:51.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:36:51.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:36:51.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:36:51.317 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:36:51.317 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:36:51.317 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:36:51.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:36:51.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:36:51.318 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:36:51.319 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:36:51.351 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:51.349+0000 7faf463667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:51.354 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:51.353+0000 7faf463667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:51.355 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:51.353+0000 7faf463667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:36:51.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:36:51.652 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:51.653 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:51.981 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:52.387 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:52.385+0000 7faf463667c0 -1 Falling back to public interface 2024-09-17T19:36:52.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:52.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:52.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:36:52.983 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:36:52.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:52.983 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:53.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:54.321 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:54.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:54.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:36:54.322 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:36:54.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:54.322 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:54.646 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:36:55.096 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:36:55.092+0000 7faf463667c0 -1 osd.0 172 log_to_monitors true 2024-09-17T19:36:55.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:36:55.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:36:55.648 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:36:55.648 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:36:55.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:36:55.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:36:56.176 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 176 up_thru 176 down_at 173 last_clean_interval [164,172) [v2:127.0.0.1:6802/1416846546,v1:127.0.0.1:6803/1416846546] [v2:127.0.0.1:6804/1416846546,v1:127.0.0.1:6805/1416846546] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:36:56.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:36:56.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:36:56.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:36:56.176 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:36:56.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:36:56.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:36:56.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:36:56.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:36:56.178 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:36:56.178 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:36:56.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:36:56.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:36:56.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:36:56.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:36:56.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:36:56.297 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:36:56.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:36:56.298 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:36:56.298 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:36:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:36:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:36:56.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:36:56.662 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:56.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:36:56.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=755914244098 2024-09-17T19:36:56.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 755914244098 2024-09-17T19:36:56.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-755914244098' 2024-09-17T19:36:56.812 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:36:56.812 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:36:56.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=730144440325 2024-09-17T19:36:56.977 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 730144440325 2024-09-17T19:36:56.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-755914244098 1-730144440325' 2024-09-17T19:36:56.978 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:56.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-755914244098 2024-09-17T19:36:56.978 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:56.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:36:56.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-755914244098 2024-09-17T19:36:56.981 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:56.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=755914244098 2024-09-17T19:36:56.982 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 755914244098' 2024-09-17T19:36:56.982 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 755914244098 2024-09-17T19:36:56.983 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:57.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 755914244098 2024-09-17T19:36:57.310 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:36:58.312 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:36:58.313 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:36:58.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 755914244098 -lt 755914244098 2024-09-17T19:36:58.647 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:36:58.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:36:58.648 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-730144440325 2024-09-17T19:36:58.649 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:36:58.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-730144440325 2024-09-17T19:36:58.650 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:36:58.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=730144440325 2024-09-17T19:36:58.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 730144440325' 2024-09-17T19:36:58.652 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 730144440325 2024-09-17T19:36:58.652 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:36:58.989 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 730144440325 -lt 730144440325 2024-09-17T19:36:58.990 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:36:58.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:58.991 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:36:59.400 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:36:59.401 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:36:59.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:36:59.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:36:59.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:36:59.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:36:59.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:36:59.402 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:36:59.741 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:36:59.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:36:59.742 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:36:59.743 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:00.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:37:00.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:37:00.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:37:00.147 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:37:00.148 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:37:00.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:37:00.149 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:37:00.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:37:00.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:37:00.149 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:37:00.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:37:00.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:37:00.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:37:00.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:37:00.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:37:00.458 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:37:00.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:37:00.459 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-09-17T19:37:01.628 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-17T19:37:01.628 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:37:02.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:37:02.188 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:37:02.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:37:02.189 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:37:02.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:37:02.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:37:02.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:37:02.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:37:02.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:37:02.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:37:02.191 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:37:02.192 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:37:02.192 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:37:02.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:37:02.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:37:02.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:37:02.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:37:02.231 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:02.228+0000 7f92de6217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:02.234 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:02.232+0000 7f92de6217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:02.236 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:02.232+0000 7f92de6217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:02.530 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:02.531 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:02.859 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:03.479 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:03.476+0000 7f92de6217c0 -1 Falling back to public interface 2024-09-17T19:37:03.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:03.860 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:03.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:37:03.861 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:37:03.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:03.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:04.187 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:05.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:37:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:37:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:05.190 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:05.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:05.977 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:05.976+0000 7f92de6217c0 -1 osd.1 178 log_to_monitors true 2024-09-17T19:37:06.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:06.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:06.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:37:06.510 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:37:06.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:06.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:06.847 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 182 up_thru 182 down_at 179 last_clean_interval [170,178) [v2:127.0.0.1:6810/1052651606,v1:127.0.0.1:6811/1052651606] [v2:127.0.0.1:6812/1052651606,v1:127.0.0.1:6813/1052651606] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:37:06.848 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:37:06.849 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:37:06.849 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:37:06.850 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:37:06.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:37:06.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:37:06.850 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:37:06.965 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:37:06.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:37:06.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:37:06.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:37:06.966 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:37:06.966 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:37:07.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:37:07.376 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:37:07.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:37:07.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:07.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:37:07.547 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=755914244101 2024-09-17T19:37:07.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 755914244101 2024-09-17T19:37:07.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-755914244101' 2024-09-17T19:37:07.548 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:07.548 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:37:07.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=781684047874 2024-09-17T19:37:07.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 781684047874 2024-09-17T19:37:07.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-755914244101 1-781684047874' 2024-09-17T19:37:07.693 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:07.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-755914244101 2024-09-17T19:37:07.694 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:07.694 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:37:07.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-755914244101 2024-09-17T19:37:07.695 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:07.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=755914244101 2024-09-17T19:37:07.696 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 755914244101' 2024-09-17T19:37:07.697 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 755914244101 2024-09-17T19:37:07.697 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:08.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 755914244100 -lt 755914244101 2024-09-17T19:37:08.002 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:37:09.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:37:09.004 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:09.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 755914244101 -lt 755914244101 2024-09-17T19:37:09.316 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:09.318 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-781684047874 2024-09-17T19:37:09.318 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:09.320 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:37:09.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-781684047874 2024-09-17T19:37:09.321 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:09.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=781684047874 2024-09-17T19:37:09.323 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 781684047874' 2024-09-17T19:37:09.323 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 781684047874 2024-09-17T19:37:09.323 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:37:09.661 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 781684047874 -lt 781684047874 2024-09-17T19:37:09.661 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:37:09.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:09.662 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:10.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:37:10.070 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:37:10.070 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:37:10.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:37:10.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:37:10.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:37:10.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:37:10.071 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:37:10.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:37:10.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:37:10.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:10.413 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:37:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' -1 2024-09-17T19:37:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-17T19:37:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=3 2024-09-17T19:37:10.776 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:37:10.777 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-17T19:37:10.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:37:10.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:37:10.878 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:37:10.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:37:10.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:37:10.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:37:10.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:37:10.879 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:37:11.203 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-17T19:37:11.203 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:37:11.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:11.204 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:11.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 6 2024-09-17T19:37:11.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 3 '!=' 3 2024-09-17T19:37:11.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-17T19:37:11.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-17T19:37:11.578 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-17T19:37:11.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-17T19:37:11.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-17T19:37:11.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-17T19:37:11.579 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-17T19:37:11.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-17T19:37:11.974 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-17T19:37:11.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 13 )) 2024-09-17T19:37:11.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-17T19:37:11.975 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-17T19:37:12.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-17T19:37:12.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:37:12.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:37:12.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:37:12.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:37:12.176 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:37:12.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:37:12.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:37:12.497 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:37:12.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:37:12.497 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:12.498 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:12.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:37:12.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:37:12.864 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:37:12.865 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:37:12.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:37:12.866 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:37:13.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:37:13.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-09-17T19:37:14.333 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-17T19:37:14.334 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:37:14.870 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:37:14.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:37:14.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:37:14.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:37:14.871 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:14.872 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:14.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:14.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:37:14.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:37:14.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:37:14.873 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:37:14.874 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:37:14.875 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:37:14.875 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:37:14.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:37:14.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:37:14.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:37:14.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:37:14.909 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:14.908+0000 7f2ec98ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:14.912 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:14.912+0000 7f2ec98ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:14.914 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:14.912+0000 7f2ec98ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:15.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:37:15.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:37:15.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:37:15.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:37:15.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:37:15.350 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:15.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:37:15.351 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:37:15.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:15.351 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:15.553 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:16.395 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:16.392+0000 7f2ec98ca7c0 -1 Falling back to public interface 2024-09-17T19:37:16.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:16.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:16.555 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:37:16.555 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:37:16.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:16.556 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:16.884 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:17.885 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:17.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:17.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:37:17.886 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:37:17.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:17.886 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:18.225 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:18.845 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:18.844+0000 7f2ec98ca7c0 -1 osd.1 184 log_to_monitors true 2024-09-17T19:37:19.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:19.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:19.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:37:19.227 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:37:19.227 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:19.228 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:19.589 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 188 up_thru 182 down_at 185 last_clean_interval [182,184) [v2:127.0.0.1:6810/2662428588,v1:127.0.0.1:6811/2662428588] [v2:127.0.0.1:6812/2662428588,v1:127.0.0.1:6813/2662428588] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:37:19.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:37:19.589 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:37:19.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:37:19.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:37:19.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:37:19.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:37:19.590 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:37:19.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:37:19.591 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:37:19.591 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:37:19.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:37:19.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:37:19.591 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:37:19.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:37:19.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:37:19.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:37:19.704 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:37:19.705 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:37:19.705 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:37:20.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:37:20.030 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:37:20.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:37:20.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:20.031 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:37:20.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=755914244104 2024-09-17T19:37:20.266 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 755914244104 2024-09-17T19:37:20.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-755914244104' 2024-09-17T19:37:20.267 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:20.267 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:37:20.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=807453851650 2024-09-17T19:37:20.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 807453851650 2024-09-17T19:37:20.408 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-755914244104 1-807453851650' 2024-09-17T19:37:20.409 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:20.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-755914244104 2024-09-17T19:37:20.409 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:20.410 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:37:20.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-755914244104 2024-09-17T19:37:20.411 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:20.411 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=755914244104 2024-09-17T19:37:20.412 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 755914244104' 2024-09-17T19:37:20.412 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 755914244104 2024-09-17T19:37:20.412 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:20.764 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 755914244103 -lt 755914244104 2024-09-17T19:37:20.765 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:37:21.766 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:37:21.767 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:22.087 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 755914244103 -lt 755914244104 2024-09-17T19:37:22.088 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:37:23.089 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:37:23.090 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:23.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 755914244105 -lt 755914244104 2024-09-17T19:37:23.421 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:23.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-807453851650 2024-09-17T19:37:23.422 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:23.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:37:23.424 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-807453851650 2024-09-17T19:37:23.425 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:23.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=807453851650 2024-09-17T19:37:23.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 807453851650' 2024-09-17T19:37:23.426 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 807453851650 2024-09-17T19:37:23.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:37:23.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 807453851650 -lt 807453851650 2024-09-17T19:37:23.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:37:23.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:23.760 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:24.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:37:24.170 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:37:24.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:37:24.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:37:24.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:37:24.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:37:24.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:37:24.171 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:37:24.492 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:37:24.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:37:24.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:24.493 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:24.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:37:24.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:37:24.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:37:24.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-09-17T19:37:24.861 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-09-17T19:37:24.861 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-09-17T19:37:24.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-09-17T19:37:24.862 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-09-17T19:37:24.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-09-17T19:37:24.862 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2024-09-17T19:37:24.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2024-09-17T19:37:24.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-09-17T19:37:24.863 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-17T19:37:25.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-09-17T19:37:25.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-09-17T19:37:25.346 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0: {} 2024-09-17T19:37:25.346 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-17T19:37:25.361 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1: {} 2024-09-17T19:37:25.362 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-17T19:37:25.376 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-17T19:37:25.377 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:37:25.672 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:37:25.673 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-17T19:37:25.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:37:25.673 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T19:37:25.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:37:25.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T19:37:25.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:37:25.674 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:37:25.674 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:37:25.675 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T19:37:25.675 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:37:25.675 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:37:25.676 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:25.676 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:25.676 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:25.676 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:37:25.677 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:37:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:37:25.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:37:25.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:37:25.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T19:37:25.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:37:25.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:37:25.801 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:25.801 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:25.801 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:25.801 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:37:25.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:37:25.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-17T19:37:25.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-17T19:37:25.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:37:25.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:37:25.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-17T19:37:25.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:37:25.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T19:37:25.925 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:37:25.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-17T19:37:25.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:37:25.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-17T19:37:25.926 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:37:25.926 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-17T19:37:26.240 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:37:26.241 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:37:26.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:37:26.244 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-17T19:37:26.244 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:37:26.244 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:37:26.245 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:26.245 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:26.245 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:26.245 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:37:26.245 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:37:26.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:37:26.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:37:26.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-17T19:37:26.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:37:26.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:37:26.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:37:26.364 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:26.364 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:26.364 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:26.364 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:37:26.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:37:26.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-17T19:37:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-17T19:37:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:37:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:37:26.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T19:37:26.485 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:37:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-17T19:37:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:37:26.486 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-17T19:37:26.486 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-17T19:37:26.486 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:37:26.797 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-17T19:37:26.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:37:26.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-17T19:37:26.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:37:26.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-17T19:37:26.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:37:26.798 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:37:26.799 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:37:26.799 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:37:26.800 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:37:26.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:37:26.923 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:37:26.924 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:37:26.924 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-17T19:37:26.925 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:37:26.925 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:37:26.925 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:26.925 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:26.925 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:26.926 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:37:26.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:37:26.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-17T19:37:27.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-17T19:37:27.047 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T19:37:27.048 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:37:27.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-17T19:37:27.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:37:27.049 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-17T19:37:27.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-17T19:37:27.049 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:37:27.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:37:27.362 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-17T19:37:27.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:37:27.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T19:37:27.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:37:27.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T19:37:27.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:37:27.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:37:27.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:37:27.365 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T19:37:27.365 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:37:27.365 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:37:27.366 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:27.366 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:27.366 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:27.366 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:37:27.366 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:37:27.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:37:27.481 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:37:27.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:37:27.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T19:37:27.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:37:27.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:37:27.483 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:37:27.484 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:27.484 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:27.484 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:37:27.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:37:27.484 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-17T19:37:27.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-09-17T19:37:27.593 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_scrub: local pgid=3.0 2024-09-17T19:37:27.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: get_last_scrub_stamp 3.0 2024-09-17T19:37:27.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:37:27.594 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:37:27.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:37:27.595 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:37:27.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_scrub: local last_scrub=2024-09-17T19:32:47.366014+0000 2024-09-17T19:37:27.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_scrub: ceph pg scrub 3.0 2024-09-17T19:37:28.168 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0 on osd.1 to scrub 2024-09-17T19:37:28.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: pg_scrub: wait_for_scrub 3.0 2024-09-17T19:32:47.366014+0000 2024-09-17T19:37:28.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:37:28.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:32:47.366014+0000 2024-09-17T19:37:28.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:37:28.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:37:28.184 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:37:28.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:37:28.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:37:28.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:37:28.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:37:28.185 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:37:28.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:47.366014+0000 '>' 2024-09-17T19:32:47.366014+0000 2024-09-17T19:37:28.501 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:37:29.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:37:29.503 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:37:29.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:37:29.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:37:29.504 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:37:29.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:37:29.505 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:37:29.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:47.366014+0000 '>' 2024-09-17T19:32:47.366014+0000 2024-09-17T19:37:29.816 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:37:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:37:30.817 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:37:30.818 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:37:30.818 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:37:30.818 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:37:30.818 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:37:30.819 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:37:28.860929+0000 '>' 2024-09-17T19:32:47.366014+0000 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-17T19:37:31.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-17T19:37:31.153 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-17T19:37:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-17T19:37:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-09-17T19:37:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-09-17T19:37:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-09-17T19:37:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.154 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.163 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.189 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.199 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.200 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.210 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.221 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.234 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.245 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.269 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.301 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.327 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.352 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.367 INFO:tasks.workunit.client.0.smithi105.stdout:Missing log message 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can't decode 'snapset' attr .* v=3 cannot decode .* Malformed input' 2024-09-17T19:37:31.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1186: TEST_corrupt_scrub_replicated: echo 'Missing log message '\''log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input'\''' 2024-09-17T19:37:31.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1187: TEST_corrupt_scrub_replicated: expr 0 + 1 2024-09-17T19:37:31.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1187: TEST_corrupt_scrub_replicated: ERRORS=1 2024-09-17T19:37:31.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.369 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.379 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.392 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:37:31.406 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-09-17T19:37:31.418 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-17T19:37:31.453 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-17T19:37:31.482 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-17T19:37:31.483 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-17T19:37:31.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-17T19:37:31.512 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-17T19:37:31.541 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-17T19:37:31.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=188 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:37:31.570 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:37:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:37:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:37:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T19:37:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-17T19:37:31.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:37:31.619 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:37:31.620 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:37:31.620 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:37:31.620 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:37:31.620 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-17T19:37:31.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-17T19:37:31.620 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:37:31.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:37:31.665 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:37:31.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-09-17T19:37:31.666 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-17T19:37:31.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-09-17T19:37:31.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-09-17T19:37:31.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:37:31.758 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:37:31.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:37:31.759 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:37:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:37:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:37:31.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:37:32.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:37:32.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:37:32.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:37:32.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:37:32.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:37:32.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:37:32.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:37:32.069 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-09-17T19:37:33.503 INFO:tasks.workunit.client.0.smithi105.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-17T19:37:34.038 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:37:34.039 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:37:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:37:34.040 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:37:34.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:37:34.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:37:34.040 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:37:34.041 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:37:34.042 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:37:34.043 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:37:34.043 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:37:34.043 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:37:34.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:37:34.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:37:34.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:37:34.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:37:34.077 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:34.076+0000 7f44c02487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:34.079 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:34.076+0000 7f44c02487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:34.081 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:34.080+0000 7f44c02487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:37:34.387 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:37:34.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:34.388 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:34.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:35.375 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:35.372+0000 7f44c02487c0 -1 Falling back to public interface 2024-09-17T19:37:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:37:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:37:35.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:35.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:36.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:37.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:37.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:37.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:37:37.073 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:37:37.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:37.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:37.391 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:38.350 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:38.348+0000 7f44c02487c0 -1 osd.0 190 log_to_monitors true 2024-09-17T19:37:38.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:38.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:38.393 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:37:38.393 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:37:38.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:38.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:38.757 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:39.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:39.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:39.759 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:37:39.759 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:37:39.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:39.760 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:40.174 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 194 up_thru 194 down_at 191 last_clean_interval [176,190) [v2:127.0.0.1:6802/1723539083,v1:127.0.0.1:6803/1723539083] [v2:127.0.0.1:6804/1723539083,v1:127.0.0.1:6805/1723539083] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:37:40.175 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:37:40.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:37:40.177 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:37:40.177 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:37:40.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:37:40.177 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:37:40.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:37:40.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:37:40.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:37:40.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:37:40.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:37:40.286 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:37:40.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:37:40.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:37:40.637 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:37:40.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:37:40.637 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:40.637 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:37:40.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=833223655426 2024-09-17T19:37:40.794 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 833223655426 2024-09-17T19:37:40.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-833223655426' 2024-09-17T19:37:40.795 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:40.795 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:37:40.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=807453851655 2024-09-17T19:37:40.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 807453851655 2024-09-17T19:37:40.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-833223655426 1-807453851655' 2024-09-17T19:37:40.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:40.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-833223655426 2024-09-17T19:37:40.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:40.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:37:40.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-833223655426 2024-09-17T19:37:40.938 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:40.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=833223655426 2024-09-17T19:37:40.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 833223655426' 2024-09-17T19:37:40.939 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 833223655426 2024-09-17T19:37:40.940 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:41.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 833223655426 2024-09-17T19:37:41.239 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:37:42.240 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:37:42.241 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:42.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 833223655426 -lt 833223655426 2024-09-17T19:37:42.579 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:42.580 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-807453851655 2024-09-17T19:37:42.580 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:42.582 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:37:42.583 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-807453851655 2024-09-17T19:37:42.583 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:42.584 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=807453851655 2024-09-17T19:37:42.585 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 807453851655' 2024-09-17T19:37:42.585 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 807453851655 2024-09-17T19:37:42.585 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:37:42.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 807453851655 -lt 807453851655 2024-09-17T19:37:42.899 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:37:42.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:42.900 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:43.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:37:43.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:37:43.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:37:43.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:37:43.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:37:43.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:37:43.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:37:43.292 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:37:43.631 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:37:43.632 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:37:43.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:43.633 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:44.004 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:37:44.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:37:44.006 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:37:44.007 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:37:44.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:37:44.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:37:44.007 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:37:44.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:37:44.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-09-17T19:37:44.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:37:44.314 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:37:44.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:37:44.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:37:44.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:37:44.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-09-17T19:37:45.381 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-17T19:37:45.381 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:37:45.921 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:37:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:37:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:37:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:37:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:37:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:37:45.935 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:37:45.936 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:37:45.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:37:45.937 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:37:45.937 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:45.937 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:45.937 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:45.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:45.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:37:45.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:37:45.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:37:45.939 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:37:45.940 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:37:45.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:37:45.941 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:37:45.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:37:45.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:37:45.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:37:45.941 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:37:45.958 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:45.955+0000 7fe3b84017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:45.961 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:45.959+0000 7fe3b84017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:45.963 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:45.959+0000 7fe3b84017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:46.260 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:37:46.261 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:37:46.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:46.261 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:46.605 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:47.051 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:47.047+0000 7fe3b84017c0 -1 Falling back to public interface 2024-09-17T19:37:47.607 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:47.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:47.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:37:47.608 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:37:47.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:47.608 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:47.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:48.929 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:48.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:48.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:37:48.930 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:37:48.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:48.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:49.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:49.508 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:49.503+0000 7fe3b84017c0 -1 osd.1 196 log_to_monitors true 2024-09-17T19:37:50.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:50.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:50.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:37:50.257 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:37:50.257 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:50.258 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:37:50.667 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 200 up_thru 200 down_at 197 last_clean_interval [188,196) [v2:127.0.0.1:6810/4283736394,v1:127.0.0.1:6811/4283736394] [v2:127.0.0.1:6812/4283736394,v1:127.0.0.1:6813/4283736394] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:37:50.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:37:50.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:37:50.667 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:37:50.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:37:50.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:37:50.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:37:50.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:37:50.668 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:37:50.668 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:37:50.669 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:37:50.669 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:37:50.669 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:37:50.669 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:37:50.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:37:50.792 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:37:50.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:37:50.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:37:50.793 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:37:50.793 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:37:51.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:37:51.263 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:37:51.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:37:51.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:51.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:37:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=833223655429 2024-09-17T19:37:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 833223655429 2024-09-17T19:37:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-833223655429' 2024-09-17T19:37:51.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:37:51.426 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:37:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=858993459202 2024-09-17T19:37:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 858993459202 2024-09-17T19:37:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-833223655429 1-858993459202' 2024-09-17T19:37:51.567 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:51.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-833223655429 2024-09-17T19:37:51.568 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:51.569 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:37:51.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-833223655429 2024-09-17T19:37:51.570 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:51.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=833223655429 2024-09-17T19:37:51.571 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 833223655429' 2024-09-17T19:37:51.572 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 833223655429 2024-09-17T19:37:51.572 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:51.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 833223655428 -lt 833223655429 2024-09-17T19:37:51.881 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:37:52.882 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:37:52.883 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:37:53.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 833223655429 -lt 833223655429 2024-09-17T19:37:53.206 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:37:53.207 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-858993459202 2024-09-17T19:37:53.208 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:37:53.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:37:53.210 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:37:53.210 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-858993459202 2024-09-17T19:37:53.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=858993459202 2024-09-17T19:37:53.211 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 858993459202' 2024-09-17T19:37:53.211 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 858993459202 2024-09-17T19:37:53.212 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:37:53.510 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 858993459202 -lt 858993459202 2024-09-17T19:37:53.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:37:53.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:53.511 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:53.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:37:53.960 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:37:53.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:37:53.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:37:53.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:37:53.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:37:53.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:37:53.961 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:37:54.315 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:37:54.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:37:54.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:37:54.316 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:37:54.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:37:54.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:37:54.716 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:37:54.717 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-09-17T19:37:54.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:37:54.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:37:54.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:37:54.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2024-09-17T19:37:54.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:37:54.831 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:37:54.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:37:54.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:37:54.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2024-09-17T19:37:54.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:37:54.832 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-09-17T19:37:54.832 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:37:54.833 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:37:54.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:37:54.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:37:54.833 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:37:55.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:37:55.143 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:37:56.802 INFO:tasks.workunit.client.0.smithi105.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-09-17T19:37:57.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-09-17T19:37:57.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:37:57.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:37:57.339 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:37:57.340 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:37:57.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:37:57.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:37:57.341 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:37:57.341 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:37:57.341 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:37:57.341 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:37:57.343 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:37:57.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:37:57.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-09-17T19:37:57.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-17T19:37:57.344 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.0 2024-09-17T19:37:57.344 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:37:57.345 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-09-17T19:37:57.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-17T19:37:57.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:37:57.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:37:57.345 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:37:57.377 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:57.375+0000 7fa3538c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:57.380 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:57.379+0000 7fa3538c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:57.382 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:57.379+0000 7fa3538c07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:37:57.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-17T19:37:57.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:37:57.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-17T19:37:57.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:37:57.668 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:37:57.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:57.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:37:57.669 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:37:57.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:57.669 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:58.003 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:37:58.851 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:37:58.847+0000 7fa3538c07c0 -1 Falling back to public interface 2024-09-17T19:37:59.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:37:59.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:37:59.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:37:59.005 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:37:59.005 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:37:59.006 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:37:59.334 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:38:00.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:38:00.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:00.336 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:38:00.337 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:38:00.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:00.337 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:38:00.677 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:38:01.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:38:01.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:01.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:38:01.680 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:38:01.680 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:01.681 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:38:01.742 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:38:01.739+0000 7fa3538c07c0 -1 osd.0 202 log_to_monitors true 2024-09-17T19:38:02.063 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:38:03.065 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:38:03.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:03.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-17T19:38:03.066 INFO:tasks.workunit.client.0.smithi105.stderr:4 2024-09-17T19:38:03.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:03.066 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 up in weight 1 up_from 206 up_thru 206 down_at 203 last_clean_interval [194,202) [v2:127.0.0.1:6802/4242086616,v1:127.0.0.1:6803/4242086616] [v2:127.0.0.1:6804/4242086616,v1:127.0.0.1:6805/4242086616] exists,up c8f8e2c8-d609-4274-a730-187075fe3e68 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:38:03.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:38:03.439 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:38:03.440 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:38:03.440 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:38:03.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:38:03.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:38:03.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:38:03.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:38:03.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:38:03.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:38:03.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:38:03.561 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:38:03.561 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:38:03.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:38:03.889 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:38:03.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:38:03.889 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:38:03.889 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:38:04.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=884763262978 2024-09-17T19:38:04.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 884763262978 2024-09-17T19:38:04.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-884763262978' 2024-09-17T19:38:04.046 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:38:04.046 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:38:04.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=858993459205 2024-09-17T19:38:04.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 858993459205 2024-09-17T19:38:04.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-884763262978 1-858993459205' 2024-09-17T19:38:04.191 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:38:04.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-884763262978 2024-09-17T19:38:04.191 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:38:04.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:38:04.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-884763262978 2024-09-17T19:38:04.194 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:38:04.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=884763262978 2024-09-17T19:38:04.195 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 884763262978' 2024-09-17T19:38:04.195 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 884763262978 2024-09-17T19:38:04.195 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:04.524 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 884763262978 2024-09-17T19:38:04.525 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:38:05.526 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:38:05.527 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:05.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 884763262978 2024-09-17T19:38:05.850 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:38:06.851 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:38:06.852 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:07.173 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 884763262978 -lt 884763262978 2024-09-17T19:38:07.174 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:38:07.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-858993459205 2024-09-17T19:38:07.175 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:38:07.177 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:38:07.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-858993459205 2024-09-17T19:38:07.178 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:38:07.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=858993459205 2024-09-17T19:38:07.179 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 858993459205' 2024-09-17T19:38:07.179 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 858993459205 2024-09-17T19:38:07.179 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:38:07.509 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 858993459206 -lt 858993459205 2024-09-17T19:38:07.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:38:07.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:38:07.510 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:38:07.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:38:07.933 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:38:07.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:38:07.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:38:07.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:38:07.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:38:07.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:38:07.934 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:38:08.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:38:08.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:38:08.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:38:08.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:38:08.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:38:08.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:38:08.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:38:08.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:38:08.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:38:08.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-17T19:38:09.044 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:38:09.045 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-09-17T19:38:10.194 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-09-17T19:38:10.194 INFO:tasks.workunit.client.0.smithi105.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-09-17T19:38:10.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-09-17T19:38:10.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2024-09-17T19:38:10.733 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-17T19:38:10.734 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:10.735 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-17T19:38:10.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-17T19:38:10.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-17T19:38:10.737 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-09-17T19:38:10.739 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-17T19:38:10.739 INFO:tasks.workunit.client.0.smithi105.stderr:start osd.1 2024-09-17T19:38:10.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=f6839024-bcc9-40d7-925b-598e9d890d00 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.46548/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-17T19:38:10.740 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-09-17T19:38:10.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-17T19:38:10.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-17T19:38:10.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-17T19:38:10.740 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-17T19:38:10.769 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:38:10.767+0000 7fb8b927c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:38:10.772 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:38:10.767+0000 7fb8b927c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:38:10.773 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:38:10.771+0000 7fb8b927c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-17T19:38:11.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-17T19:38:11.075 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:0 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:38:11.076 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:11.423 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:38:12.091 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:38:12.087+0000 7fb8b927c7c0 -1 Falling back to public interface 2024-09-17T19:38:12.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:38:12.425 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:12.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-17T19:38:12.426 INFO:tasks.workunit.client.0.smithi105.stderr:1 2024-09-17T19:38:12.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:12.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:38:12.748 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:38:13.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:38:13.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:13.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-17T19:38:13.750 INFO:tasks.workunit.client.0.smithi105.stderr:2 2024-09-17T19:38:13.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:13.750 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:38:14.072 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-17T19:38:14.704 INFO:tasks.workunit.client.0.smithi105.stderr:2024-09-17T19:38:14.699+0000 7fb8b927c7c0 -1 osd.1 208 log_to_monitors true 2024-09-17T19:38:15.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-17T19:38:15.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-17T19:38:15.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-17T19:38:15.074 INFO:tasks.workunit.client.0.smithi105.stderr:3 2024-09-17T19:38:15.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-17T19:38:15.074 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-17T19:38:15.506 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1 up in weight 1 up_from 212 up_thru 206 down_at 209 last_clean_interval [200,208) [v2:127.0.0.1:6810/309953917,v1:127.0.0.1:6811/309953917] [v2:127.0.0.1:6812/309953917,v1:127.0.0.1:6813/309953917] exists,up ece21a65-749d-4e73-a3c1-8b3d29b5ecb6 2024-09-17T19:38:15.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-17T19:38:15.506 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-17T19:38:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-17T19:38:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2024-09-17T19:38:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:38:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:38:15.507 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:38:15.507 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:38:15.508 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:38:15.508 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:38:15.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:38:15.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:38:15.509 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:38:15.626 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:38:15.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:38:15.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:38:15.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:38:15.627 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:38:15.627 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:38:16.029 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:38:16.029 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:38:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:38:16.030 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:38:16.030 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:38:16.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=884763262981 2024-09-17T19:38:16.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 884763262981 2024-09-17T19:38:16.217 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-884763262981' 2024-09-17T19:38:16.218 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:38:16.218 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:38:16.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=910533066754 2024-09-17T19:38:16.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 910533066754 2024-09-17T19:38:16.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-884763262981 1-910533066754' 2024-09-17T19:38:16.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:38:16.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-884763262981 2024-09-17T19:38:16.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:38:16.364 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:38:16.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-884763262981 2024-09-17T19:38:16.365 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:38:16.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=884763262981 2024-09-17T19:38:16.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 884763262981' 2024-09-17T19:38:16.366 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.0 seq 884763262981 2024-09-17T19:38:16.366 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:16.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 884763262980 -lt 884763262981 2024-09-17T19:38:16.753 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:38:17.689 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-17T19:38:17.689 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:18.020 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 884763262980 -lt 884763262981 2024-09-17T19:38:18.021 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-17T19:38:19.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-17T19:38:19.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:19.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 884763262982 -lt 884763262981 2024-09-17T19:38:19.357 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:38:19.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-910533066754 2024-09-17T19:38:19.358 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:38:19.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:38:19.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-910533066754 2024-09-17T19:38:19.361 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:38:19.362 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=910533066754 2024-09-17T19:38:19.363 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 910533066754' 2024-09-17T19:38:19.363 INFO:tasks.workunit.client.0.smithi105.stderr:waiting osd.1 seq 910533066754 2024-09-17T19:38:19.363 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:38:19.729 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 910533066754 -lt 910533066754 2024-09-17T19:38:19.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:38:19.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:38:19.730 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:38:20.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:38:20.120 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:38:20.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:38:20.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:38:20.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:38:20.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:38:20.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:38:20.121 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:38:20.426 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:38:20.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:38:20.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:38:20.427 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:38:20.807 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:38:20.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:38:20.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:38:20.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:38:20.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ11 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ11 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2024-09-17T19:38:20.811 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-09-17T19:38:20.812 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:38:21.141 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:38:21.142 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-17T19:38:21.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:38:21.144 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:38:21.145 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:21.146 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:21.146 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:21.146 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:38:21.146 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:38:21.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:38:21.252 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:38:21.253 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:38:21.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T19:38:21.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:38:21.253 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:38:21.254 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:21.255 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:21.255 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:21.255 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:38:21.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:38:21.255 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-09-17T19:38:21.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-09-17T19:38:21.365 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ12 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:38:21.366 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:38:21.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T19:38:21.367 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:38:21.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ12 2024-09-17T19:38:21.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:38:21.367 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2024-09-17T19:38:21.368 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-09-17T19:38:21.368 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:38:21.683 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:38:21.684 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:38:21.687 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:38:21.688 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:38:21.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:38:21.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:38:21.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:38:21.806 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-17T19:38:21.806 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:38:21.806 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:38:21.807 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:21.807 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:21.808 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:21.808 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:38:21.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:38:21.808 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-09-17T19:38:21.926 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=mdata 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=1 2024-09-17T19:38:21.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:38:21.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-17T19:38:21.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:38:21.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-17T19:38:21.929 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-17T19:38:21.929 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:38:22.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:38:22.277 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:38:22.277 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:38:22.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:38:22.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:38:22.277 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=0 2024-09-17T19:38:22.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:38:22.278 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:38:22.278 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/0/type 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:38:22.279 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:38:22.281 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:38:22.394 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:38:22.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:38:22.395 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:38:22.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.0 2024-09-17T19:38:22.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-17T19:38:22.396 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-17T19:38:22.397 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:22.397 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:22.397 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:22.397 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.0.asok 2024-09-17T19:38:22.397 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:38:22.398 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=csr_pool 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-17T19:38:22.517 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=ROBJ13 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/osd-scrub-repair 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds csr_pool ROBJ13 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2024-09-17T19:38:22.518 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2024-09-17T19:38:22.519 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-09-17T19:38:22.519 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr:0' 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0') 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-17T19:38:22.839 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/osd-scrub-repair/1/type 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-17T19:38:22.841 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-17T19:38:22.842 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-17T19:38:22.842 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-17T19:38:22.842 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:38:22.842 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:38:22.843 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:22.843 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:22.843 INFO:tasks.workunit.client.0.smithi105.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:22.843 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:38:22.844 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.46548/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-17T19:38:22.950 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-17T19:38:22.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-17T19:38:22.951 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-17T19:38:22.952 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.46548/ceph-osd.1.asok 2024-09-17T19:38:22.953 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-17T19:38:22.954 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.46548/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-09-17T19:38:23.068 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-09-17T19:38:23.194 INFO:tasks.workunit.client.0.smithi105.stdout:osd.0: {} 2024-09-17T19:38:23.195 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0: osd_max_object_size = '' 2024-09-17T19:38:23.210 INFO:tasks.workunit.client.0.smithi105.stdout:osd.1: {} 2024-09-17T19:38:23.210 INFO:tasks.workunit.client.0.smithi105.stderr:osd.1: osd_max_object_size = '' 2024-09-17T19:38:23.222 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-09-17T19:38:23.223 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: pg_deep_scrub: local pgid=3.0 2024-09-17T19:38:23.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:38:23.223 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:23.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:38:23.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:23.224 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:38:23.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: pg_deep_scrub: local last_scrub=2024-09-17T19:32:47.366014+0000 2024-09-17T19:38:23.520 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-09-17T19:38:23.804 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-09-17T19:38:23.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: pg_deep_scrub: wait_for_scrub 3.0 2024-09-17T19:32:47.366014+0000 last_deep_scrub_stamp 2024-09-17T19:38:23.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:38:23.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:32:47.366014+0000 2024-09-17T19:38:23.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-09-17T19:38:23.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:38:23.820 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:38:23.821 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:38:23.821 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:23.821 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:38:23.821 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:23.821 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:38:24.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:47.366014+0000 '>' 2024-09-17T19:32:47.366014+0000 2024-09-17T19:38:24.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:38:25.113 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:38:25.114 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:38:25.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:38:25.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:25.115 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:38:25.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:25.116 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:38:25.444 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:32:47.366014+0000 '>' 2024-09-17T19:32:47.366014+0000 2024-09-17T19:38:25.445 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:38:26.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:38:26.446 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:38:26.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-09-17T19:38:26.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:26.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-09-17T19:38:26.447 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:26.448 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:38:24.848001+0000 '>' 2024-09-17T19:32:47.366014+0000 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-09-17T19:38:26.772 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-09-17T19:38:26.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-09-17T19:38:26.774 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.775 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.780 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.789 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.810 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.819 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.828 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.837 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.846 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.847 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.857 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.867 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.876 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.877 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.888 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.898 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.907 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.908 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.927 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.937 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.949 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.958 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.969 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.980 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:26.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:26.991 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.001 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.012 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.025 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.036 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.049 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.061 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.073 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.085 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.099 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.111 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.137 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.152 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.167 INFO:tasks.workunit.client.0.smithi105.stdout:Missing log message 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can't decode 'snapset' attr .* v=3 cannot decode .* Malformed input' 2024-09-17T19:38:27.167 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2164: TEST_corrupt_scrub_replicated: echo 'Missing log message '\''log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input'\''' 2024-09-17T19:38:27.167 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2165: TEST_corrupt_scrub_replicated: expr 1 + 1 2024-09-17T19:38:27.168 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2165: TEST_corrupt_scrub_replicated: ERRORS=2 2024-09-17T19:38:27.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.169 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.180 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.193 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-09-17T19:38:27.194 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-09-17T19:38:27.209 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-09-17T19:38:27.243 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-09-17T19:38:27.273 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-09-17T19:38:27.273 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-09-17T19:38:27.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-09-17T19:38:27.302 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-09-17T19:38:27.331 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-09-17T19:38:27.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=212 2024-09-17T19:38:27.360 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-17T19:38:27.361 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: . as $in 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: | if type == "object" then 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: reduce keys[] as $key 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: elif type == "array" then map( walk(f) ) | f 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: else f 2024-09-17T19:38:27.414 INFO:tasks.workunit.client.0.smithi105.stderr: end; 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.version) else . end) 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-09-17T19:38:27.415 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-09-17T19:38:27.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:38:27.468 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-09-17T19:38:27.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-09-17T19:38:27.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-09-17T19:38:27.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-09-17T19:38:27.469 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: repair: local pgid=3.0 2024-09-17T19:38:27.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: get_last_scrub_stamp 3.0 2024-09-17T19:38:27.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:27.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:38:27.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:27.470 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:38:27.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1847: repair: local last_scrub=2024-09-17T19:38:24.848001+0000 2024-09-17T19:38:27.784 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: repair: ceph pg repair 3.0 2024-09-17T19:38:28.094 INFO:tasks.workunit.client.0.smithi105.stderr:instructing pg 3.0 on osd.1 to repair 2024-09-17T19:38:28.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: repair: wait_for_scrub 3.0 2024-09-17T19:38:24.848001+0000 2024-09-17T19:38:28.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: local pgid=3.0 2024-09-17T19:38:28.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: wait_for_scrub: local last_scrub=2024-09-17T19:38:24.848001+0000 2024-09-17T19:38:28.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: wait_for_scrub: local sname=last_scrub_stamp 2024-09-17T19:38:28.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i=0 )) 2024-09-17T19:38:28.112 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:38:28.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:38:28.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:28.113 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:38:28.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:28.114 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:38:28.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:38:24.848001+0000 '>' 2024-09-17T19:38:24.848001+0000 2024-09-17T19:38:28.438 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:38:29.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:38:29.439 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:38:29.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:38:29.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:29.440 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:38:29.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:29.441 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:38:29.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:38:24.848001+0000 '>' 2024-09-17T19:38:24.848001+0000 2024-09-17T19:38:29.782 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: wait_for_scrub: sleep 1 2024-09-17T19:38:30.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i++ )) 2024-09-17T19:38:30.783 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: wait_for_scrub: (( i < 300 )) 2024-09-17T19:38:30.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-09-17T19:38:30.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2024-09-17T19:38:30.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-17T19:38:30.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-17T19:38:30.785 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-09-17T19:38:31.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: wait_for_scrub: test 2024-09-17T19:38:29.906212+0000 '>' 2024-09-17T19:38:24.848001+0000 2024-09-17T19:38:31.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: wait_for_scrub: return 0 2024-09-17T19:38:31.123 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: repair: sleep 2 2024-09-17T19:38:33.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-09-17T19:38:33.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-17T19:38:33.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-17T19:38:33.125 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-17T19:38:33.126 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-17T19:38:33.127 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-17T19:38:33.127 INFO:tasks.workunit.client.0.smithi105.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-17T19:38:33.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-17T19:38:33.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-17T19:38:33.127 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-17T19:38:33.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-17T19:38:33.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-17T19:38:33.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-17T19:38:33.263 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-17T19:38:33.264 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-17T19:38:33.264 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-17T19:38:33.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-17T19:38:33.603 INFO:tasks.workunit.client.0.smithi105.stderr:1' 2024-09-17T19:38:33.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-17T19:38:33.603 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:38:33.603 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-17T19:38:33.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=884763262986 2024-09-17T19:38:33.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 884763262986 2024-09-17T19:38:33.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-884763262986' 2024-09-17T19:38:33.773 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-17T19:38:33.774 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-17T19:38:33.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=910533066758 2024-09-17T19:38:33.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 910533066758 2024-09-17T19:38:33.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-884763262986 1-910533066758' 2024-09-17T19:38:33.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:38:33.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-884763262986 2024-09-17T19:38:33.918 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:38:33.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-17T19:38:33.921 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-884763262986 2024-09-17T19:38:33.921 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:38:33.922 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.0 seq 884763262986 2024-09-17T19:38:33.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=884763262986 2024-09-17T19:38:33.922 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 884763262986' 2024-09-17T19:38:33.922 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-17T19:38:34.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 884763262986 -lt 884763262986 2024-09-17T19:38:34.285 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-17T19:38:34.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-910533066758 2024-09-17T19:38:34.286 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-17T19:38:34.289 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-17T19:38:34.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-910533066758 2024-09-17T19:38:34.289 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-17T19:38:34.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=910533066758 2024-09-17T19:38:34.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 910533066758' 2024-09-17T19:38:34.291 INFO:tasks.workunit.client.0.smithi105.stdout:waiting osd.1 seq 910533066758 2024-09-17T19:38:34.291 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-17T19:38:34.622 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 910533066758 -lt 910533066758 2024-09-17T19:38:34.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-17T19:38:34.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:38:34.623 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:38:35.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 6 == 0 2024-09-17T19:38:35.022 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-17T19:38:35.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-17T19:38:35.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-17T19:38:35.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-17T19:38:35.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-17T19:38:35.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-17T19:38:35.023 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-17T19:38:35.356 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=6 2024-09-17T19:38:35.356 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-17T19:38:35.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-17T19:38:35.357 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-17T19:38:35.735 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 = 6 2024-09-17T19:38:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-17T19:38:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-17T19:38:35.736 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-17T19:38:35.771 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-17T19:38:35.799 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-17T19:38:35.800 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-09-17T19:38:35.801 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-17T19:38:35.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-09-17T19:38:35.802 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 2 '!=' 0 ']' 2024-09-17T19:38:35.803 INFO:tasks.workunit.client.0.smithi105.stdout:TEST FAILED WITH 2 ERRORS 2024-09-17T19:38:35.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3584: TEST_corrupt_scrub_replicated: echo 'TEST FAILED WITH 2 ERRORS' 2024-09-17T19:38:35.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3585: TEST_corrupt_scrub_replicated: return 1 2024-09-17T19:38:35.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-09-17T19:38:35.804 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-scrub-repair 1 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-17T19:38:35.805 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-17T19:38:35.912 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-17T19:38:35.912 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-17T19:38:35.913 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-17T19:38:35.913 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-17T19:38:35.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-17T19:38:35.914 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-17T19:38:35.915 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-17T19:38:35.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:38:35.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-17T19:38:35.916 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-17T19:38:35.916 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T19:38:35.917 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:38:35.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-17T19:38:35.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-17T19:38:35.918 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-17T19:38:35.919 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log /home/ubuntu/cephtest/archive/log 2024-09-17T19:38:35.920 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2024-09-17T19:38:35.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-17T19:38:35.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-17T19:38:35.928 INFO:tasks.workunit.client.0.smithi105.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.46548 2024-09-17T19:38:35.928 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.46548 2024-09-17T19:38:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T19:38:35.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-17T19:38:35.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-17T19:38:35.930 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-17T19:38:35.931 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-09-17T19:38:35.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-17T19:38:36.253 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-09-17T19:38:36.426 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=58f3eb981a2d4b0d93e409f9306254d7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_1889a165b5128787ec693450a3d1c5df5d6dc203/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-09-17T19:38:36.429 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T19:38:36.437 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T19:38:36.438 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T19:38:36.478 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T19:38:36.486 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T19:38:36.486 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T19:38:36.510 INFO:teuthology.orchestra.run.smithi105.stdout: remote refid st t when poll reach delay offset jitter 2024-09-17T19:38:36.510 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================== 2024-09-17T19:38:36.511 INFO:teuthology.orchestra.run.smithi105.stdout:*hv01.front.sepi 67.205.162.81 3 u 42 256 377 0.110 -3.475 1.396 2024-09-17T19:38:36.511 INFO:teuthology.orchestra.run.smithi105.stdout:+hv02.front.sepi 63.231.80.2 3 u 80 256 377 0.109 -5.240 1.429 2024-09-17T19:38:36.511 INFO:teuthology.orchestra.run.smithi105.stdout:+hv03.front.sepi 74.6.168.72 3 u 208 256 377 0.117 -3.612 1.125 2024-09-17T19:38:36.511 INFO:teuthology.orchestra.run.smithi105.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-17T19:38:36.512 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T19:38:36.520 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T19:38:36.520 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T19:38:36.528 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T19:38:36.535 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T19:38:36.541 INFO:teuthology.task.internal:Duration was 5282.790414 seconds 2024-09-17T19:38:36.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T19:38:36.549 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T19:38:36.549 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T19:38:36.589 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T19:38:36.590 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-09-17T19:38:36.590 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T19:38:36.640 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T19:38:36.641 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T19:38:36.721 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T19:38:36.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T19:38:36.817 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T19:38:36.825 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T19:38:36.826 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T19:38:36.836 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T19:38:36.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:38:36.889 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-09-17T19:38:36.915 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T19:38:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T19:38:36.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T19:38:36.969 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T19:38:36.970 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-17_15:01:33-rados-wip-bharath10-testing-2024-09-17-1348-distro-default-smithi/7909236/remote/smithi105 2024-09-17T19:38:36.971 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T19:38:40.292 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T19:38:40.292 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T19:38:40.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T19:38:40.373 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T19:38:40.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T19:38:40.381 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T19:38:40.381 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T19:38:40.387 INFO:teuthology.orchestra.run.smithi105.stdout: 658463 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 17 19:38 /home/ubuntu/cephtest 2024-09-17T19:38:40.388 INFO:teuthology.orchestra.run.smithi105.stdout: 788943 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 17 18:20 /home/ubuntu/cephtest/mnt.0 2024-09-17T19:38:40.388 INFO:teuthology.orchestra.run.smithi105.stdout: 799258 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 17 18:27 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-17T19:38:40.388 INFO:teuthology.orchestra.run.smithi105.stdout: 935734 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 17 18:27 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-17T19:38:40.388 INFO:teuthology.orchestra.run.smithi105.stdout: 935735 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 17 19:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-17T19:38:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T19:38:40.389 INFO:teuthology.orchestra.run.smithi105.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-17T19:38:40.389 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi105 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-17T19:38:40.393 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T19:38:40.403 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T19:38:40.411 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi105 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-17T19:38:40.414 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 5282.790413618088 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi105 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=58f3eb981a2d4b0d93e409f9306254d7 status: fail success: false 2024-09-17T19:38:40.414 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T19:38:40.500 INFO:teuthology.run:FAIL